We’d like to announce the first release candidate of the Elastic Stack 8.0.0 and the continuation of the Pioneer Program. Give it a try and please share all the bugs you find on GitHub.

genericrelease.png

Elastic Stack 8.0.0-rc1 released

audit-letters-full-bleed.jpg

audit-clipboard-cc.jpg

Introducing Auditbeat: Ship Linux Audit Logs to Elasticsearch and More

container-loading-small.jpg

container-ship-rena-small.jpg

Monitoring Container Resource Usage with Metricbeat

How to use the Winlogbeat and Kibana to visualize logon events from Windows event logs.

winlogbeat-rdp-map-full-bleed.jpg

winlogbeat-account-usage-dashboard.jpg

Monitoring Windows Logons with Winlogbeat

<html><head></head><body><p><strong>Updated January 12, 2017:</strong> This post was updated to reflect changes in Packetbeat 5.x and Elasticsearch 5.x.
</p>


<p>Data observed from monitoring DNS traffic on a network can be used as an
indicator of compromise (IOC). This blog post will discuss how Elasticsearch and
Watcher can be used with Packetbeat to alert when possible malware activity is
detected.
</p>
<p><a href="/products/beats/packetbeat"><em>Packetbeat</em></a> is our open
source packet analyzer. It monitors the traffic on your network and indexes the
DNS requests and responses into Elasticsearch where aggregations can be used to
help make sense of the data.
</p>
<p><a href="/products/x-pack/alerting"><em>Alerting</em></a> (formerly <em>Watcher</em>)
is part of 
	<a href="/products/x-pack">X-Pack</a> and it provides alerting and
notifications based on changes in your data.
</p>
<p>There are many use cases for alerting on data collected by Packetbeat such as
alerting when the response times for web requests are above a threshold or when
there is spike in HTTP errors returned by your web servers. The alerting
described in this article has applications in network security. We are going to
look at one specific use case -- detecting data exfiltration over DNS tunnels.
</p>
<h2>Detecting DNS Tunnels</h2>
<p>Tunnels can be established over the DNS protocol to covertly move data or
provide a command and control channel for malware. Often this technique is used
to bypass the protections of corporate firewalls and proxy servers. Tunneling
works by encoding data in DNS requests and responses. The client issues a query
for a hostname and that query is eventually forwarded to the authoritative name
server associated with the domain.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2d58208ba076e29b7d/download?uid=bltf505dadaa68339b8" data-sys-asset-uid="bltf505dadaa68339b8" alt="Packetbeat Deployment Architecture">
</p>
<p>There are a lot of different techniques that can be employed for detecting
such traffic. We are going to look at using the number of unique hostnames for a
domain as an IOC. DNS tunneling utilities must use a new hostname for each
request which leads to a much higher number of hostnames present for the
malicious domains in comparison to legitimate domains.
</p>
<h2>Packetbeat Setup</h2>
<p>The first step is to install Packetbeat and configure it to collect DNS
traffic. For this setup, the server running Packetbeat is connected to a port
mirror so that Packetbeat can observe all the traffic between the local network
and the Internet. The Packetbeat documentation has a great 
	<a href="/guide/en/beats/packetbeat/current/packetbeat-getting-started.html">Getting
Started guide
	</a> that explains installation and setup procedure, so I will just
show the configuration used.
</p>
<pre class="prettyprint"># /etc/packetbeat/packetbeat.yml
packetbeat.interfaces.device: en0
packetbeat.protocols.dns:
  ports: [53]
  include_authorities: true
  include_additionals: true
output.elasticsearch:
  hosts: [&quot;localhost:9200&quot;]
</pre>
<h2>Watch your DNS Traffic</h2>
<p>The complete
	<a href="https://github.com/elastic/examples/blob/master/Security%20Analytics/dns_tunnel_detection/unique_hostnames_watch.json">
	watch file</a> is stored in our
	<a href="https://github.com/elastic/examples/tree/master/Security%20Analytics/dns_tunnel_detection">elastic/examples</a>
	repository along with all of the supporting files shown here. We are going to
walk through the creation of this watch step-by-step.
</p>
<h2>Watch Trigger</h2>
<p>The watch trigger specifies when the execution should start. This watch is
scheduled to execute every 15 minutes.
</p>
<pre class="prettyprint">      &quot;trigger&quot;: {
         &quot;schedule&quot;: {
            &quot;interval&quot;: &quot;15m&quot;
         }
      },
</pre>
<h2>Watch Input</h2>
<p>This first step in creating this watch is to design a set of aggregations to
be used as the input to the watch. We want to find the cardinality of the
hostnames associated with each second-level domain
(e.g. 
	<code>badguy.co.</code>).
</p>
<p>We start with a query that has just two components, a time window and a
whitelist. The time window and whitelist can be customized. Find more on this in
the 
	<a href="#tuning">Tuning the Detector</a> section.
</p>
<p>Next we use a terms aggregation to create buckets for each second-level
domain. Then we apply a sub-aggregation to get the cardinality of the hostnames
within that bucket. Finally we apply a bucket selector aggregation to select
only the buckets having more than 200 unique hostnames. The watch will generate
an alert when the number of unique hostnames breaks this threshold.
</p>
<pre class="prettyprint">GET packetbeat-*/dns/_search
{
  &quot;query&quot;: {
    &quot;bool&quot;: {
      &quot;filter&quot;: {
        &quot;range&quot;: {
          &quot;@timestamp&quot;: {
            &quot;from&quot;: &quot;now-4h&quot;
          }
        }
      },
      &quot;must_not&quot;: {
        &quot;terms&quot;: {
          &quot;dns.question.etld_plus_one&quot;: [
            &quot;akadns.net.&quot;,
            &quot;amazonaws.com.&quot;,
            &quot;apple.com.&quot;,
            &quot;apple-dns.net.&quot;,
            &quot;cloudfront.net.&quot;,
            &quot;icloud.com.&quot;,
            &quot;in-addr.arpa.&quot;,
            &quot;google.com.&quot;,
            &quot;yahoo.com.&quot;
          ]
        }
      }
    }
  },
  &quot;size&quot;: 0,
  &quot;aggs&quot;: {
    &quot;by_domain&quot;: {
      &quot;terms&quot;: {
        &quot;size&quot;: 1000,
        &quot;field&quot;: &quot;dns.question.etld_plus_one&quot;
      },
      &quot;aggs&quot;: {
        &quot;unique_hostnames&quot;: {
          &quot;cardinality&quot;: {
            &quot;field&quot;: &quot;dns.question.name&quot;
          }
        },
        &quot;total_bytes_in&quot;: {
          &quot;sum&quot;: {
            &quot;field&quot;: &quot;bytes_in&quot;
          }
        },
        &quot;total_bytes_out&quot;: {
          &quot;sum&quot;: {
            &quot;field&quot;: &quot;bytes_out&quot;
          }
        },
        &quot;high_num_hostnames&quot;: {
          &quot;bucket_selector&quot;: {
            &quot;buckets_path&quot;: {
              &quot;unique_hostnames&quot;: &quot;unique_hostnames&quot;
            },
            &quot;script&quot;: &quot;params.unique_hostnames &gt; 200&quot;
          }
        }
      }
    }
  }
}
</pre>
<p>The query above relies on the <code>dns.question.etld_plus_one</code> field provided
by Packetbeat 5.x to bucket all requests for a single domain. Packetbeat
creates this field using an embedded copy of the 
	<a href="https://publicsuffix.org/">
	Public Suffix List</a>.
</p>
<h2>Watch Condition</h2>
<p>The watch condition is what determines if an alert is triggered. The
condition here is simple. This says that if any buckets were returned then
trigger the alert.
</p>
<pre class="prettyprint">      &quot;condition&quot;: {
         &quot;script&quot;: {
            &quot;inline&quot;: &quot;ctx.payload.aggregations.by_domain.buckets.size() &gt; 0&quot;
         }
      },
</pre>
<h2>Watch Actions</h2>
<p>The watch actions are executed after the condition is met, and define the
&quot;output&quot; of a watch. For this watch we are sending an email and also writing a
message to the Elasticsearch log. A&#xA0;
	<em>transform</em>&#xA0;script is being
used to manipulate the data so that it renders better in an email.
</p>
<pre class="prettyprint">        &quot;transform&quot;: {
           &quot;script&quot;: {
              &quot;file&quot;: &quot;dns_transform&quot;
           }
        },
        &quot;actions&quot;: {
           &quot;log_domains&quot;: {
              &quot;logging&quot;: {
                 &quot;text&quot;: &quot;The following domain(s) have a high number of unique hostnames: {{ctx.payload.alerts}}&quot;
              }
           },
           &quot;email_alert&quot; : {
            &quot;email&quot;: {
              &quot;to&quot;: &quot;&apos;John Doe &lt;john.doe@example.com&gt;&apos;&quot;,
              &quot;subject&quot;: &quot;Suspected DNS Tunnel Alert&quot;,
              &quot;body&quot;: &quot;The following domain(s) have a high number of unique hostnames: {{ctx.payload.alerts}}&quot;
            }
          }
        }
</pre>
<p>Below is the dns_transform Painless script. It should be placed into the
config/scripts directory of Elasticsearch.
</p>
<pre class="prettyprint">// File: config/scripts/dns_transform.painless
def alerts = ctx.payload.aggregations.by_domain.buckets.stream().collect(Collectors.toMap(p-&gt;p.key,item-&gt;[
        &quot;total_requests&quot; : item.doc_count,
        &quot;unique_hostnames&quot; : item.unique_hostnames.value,
        &quot;total_bytes_in&quot; : item.total_bytes_in.value,
        &quot;total_bytes_out&quot; : item.total_bytes_out.value,
        &quot;total_bytes&quot; : item.total_bytes_in.value + item.total_bytes_out.value
]));
return [&quot;alerts&quot;:alerts];
</pre>
<p>Here is a sample alert. Notice it contains a some additional metrics that can
be used to gauge the severity of the situation.
</p>
<pre class="prettyprint">Date: Fri, 16 Feb 2016 11:00:01 -0500 (EST)
From: Watcher &lt;watcher@example.com&gt;
Message-Id: &lt;201602161600.u0SG01ks024814@example.com&gt;
To: John Doe &lt;john.doe@example.com&gt;
Subject: Suspected DNS Tunnel Alert
The following domain(s) have a high number of unique hostnames:
{badguy.co.={total_requests=222, unique_hostnames=222, total_bytes_in=16716.0,
total_bytes_out=35161.0, total_bytes=51877.0}}
</pre>
<p>When this alert is received, the recipient can take the domain and do a
search using the Discover application in Kibana to find the network clients
responsible for the tunnel.
</p>
<h2>Testing and Results</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2dd8edd48f7693791e/download?uid=blt8aad375573dc7783" data-sys-asset-uid="blt8aad375573dc7783" alt="Unique FQDNs per Second Level Domain">
</p>
<p>This chart shows the top ten domains with the highest number of unique
hostnames over a period of 4 hours. This data was collected from a network with
about 100 devices. During that time window I replayed a network capture
containing a tunnel created by&#xA0;
	<a href="http://code.kryo.se/iodine/">iodine</a>. The tunnel which is operating
under the fictitious domain&#xA0;
	<code>pirate.sea</code>&#xA0;was up for just 20
seconds, and yet it has the highest number of domains.
</p>
<p>If the whitelist from the watch is applied then the tunnel really stands out
among the other domains as seen below.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2d12798d8e381ff0be/download?uid=bltf1e4c3e0ba99f734" data-sys-asset-uid="bltf1e4c3e0ba99f734" alt="Unique FQDNs per Second Level Domain with Whitelist">
</p>
<h2 id="tuning">Tuning the Detector</h2>
<p>There are two variables that can be tuned -- the time window and the unique
hostnames threshold. A smaller time window can be used with a smaller threshold
to make the watch more sensitive to short duration tunnels. In a shorter time
window, domains not being used for tunneling will generally accumulate fewer
unique hostnames.
</p>
<p>Tunnels that move data slowly can be detected using a larger time window. But
using a larger time window means that valid domains with a lot of unique
hostnames, such as CDNs, will cause false positives. So if you use a large time
window you will likely need to add domains to the query&apos;s whitelist.
</p>
<h2>Conclusion</h2>
<p>It was fun combining Packetbeat and Watcher to look for DNS tunnels. Remember
&quot;defense in depth&quot; if you implement a solution like this. It is important to
layer your defenses so that if one layer fails there is another one in place to
detect.
</p></body></html>

tunnel-in-gaza.jpg

brick-tunnel.jpg

Detecting DNS Tunnels with Packetbeat and Watcher

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Software Engineer

Andrew Kroh

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.</p>

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Andrew Kroh

Software Engineer, Elastic

Elastic Stack 8.0.0-rc1 released

Introducing Auditbeat: Ship Linux Audit Logs to Elasticsearch and More

Monitoring Container Resource Usage with Metricbeat

Monitoring Windows Logons with Winlogbeat

Detecting DNS Tunnels with Packetbeat and Watcher