We’d like to announce the first release candidate of the Elastic Stack 8.0.0 and the continuation of the Pioneer Program. Give it a try and please share all the bugs you find on GitHub.

genericrelease.png

Elastic Stack 8.0.0-rc1 released

audit-letters-full-bleed.jpg

audit-clipboard-cc.jpg

Introducing Auditbeat: Ship Linux Audit Logs to Elasticsearch and More

container-loading-small.jpg

container-ship-rena-small.jpg

Monitoring Container Resource Usage with Metricbeat

How to use the Winlogbeat and Kibana to visualize logon events from Windows event logs.

winlogbeat-rdp-map-full-bleed.jpg

winlogbeat-account-usage-dashboard.jpg

Monitoring Windows Logons with Winlogbeat

<html><head></head><body><p><strong>Updated January 12, 2017:</strong> This post was updated to reflect changes in Packetbeat 5.x and Elasticsearch 5.x.
</p>


<p>Data observed from monitoring DNS traffic on a network can be used as an
indicator of compromise (IOC). This blog post will discuss how Elasticsearch and
Watcher can be used with Packetbeat to alert when possible malware activity is
detected.
</p>
<p><a href="/products/beats/packetbeat"><em>Packetbeat</em></a> is our open
source packet analyzer. It monitors the traffic on your network and indexes the
DNS requests and responses into Elasticsearch where aggregations can be used to
help make sense of the data.
</p>
<p><a href="/products/x-pack/alerting"><em>Alerting</em></a> (formerly <em>Watcher</em>)
is part of 
	<a href="/products/x-pack">X-Pack</a> and it provides alerting and
notifications based on changes in your data.
</p>
<p>There are many use cases for alerting on data collected by Packetbeat such as
alerting when the response times for web requests are above a threshold or when
there is spike in HTTP errors returned by your web servers. The alerting
described in this article has applications in network security. We are going to
look at one specific use case -- detecting data exfiltration over DNS tunnels.
</p>
<h2>Detecting DNS Tunnels</h2>
<p>Tunnels can be established over the DNS protocol to covertly move data or
provide a command and control channel for malware. Often this technique is used
to bypass the protections of corporate firewalls and proxy servers. Tunneling
works by encoding data in DNS requests and responses. The client issues a query
for a hostname and that query is eventually forwarded to the authoritative name
server associated with the domain.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2d58208ba076e29b7d/download?uid=bltf505dadaa68339b8" data-sys-asset-uid="bltf505dadaa68339b8" alt="Packetbeat Deployment Architecture">
</p>
<p>There are a lot of different techniques that can be employed for detecting
such traffic. We are going to look at using the number of unique hostnames for a
domain as an IOC. DNS tunneling utilities must use a new hostname for each
request which leads to a much higher number of hostnames present for the
malicious domains in comparison to legitimate domains.
</p>
<h2>Packetbeat Setup</h2>
<p>The first step is to install Packetbeat and configure it to collect DNS
traffic. For this setup, the server running Packetbeat is connected to a port
mirror so that Packetbeat can observe all the traffic between the local network
and the Internet. The Packetbeat documentation has a great 
	<a href="/guide/en/beats/packetbeat/current/packetbeat-getting-started.html">Getting
Started guide
	</a> that explains installation and setup procedure, so I will just
show the configuration used.
</p>
<pre class="prettyprint"># /etc/packetbeat/packetbeat.yml
packetbeat.interfaces.device: en0
packetbeat.protocols.dns:
  ports: [53]
  include_authorities: true
  include_additionals: true
output.elasticsearch:
  hosts: [&quot;localhost:9200&quot;]
</pre>
<h2>Watch your DNS Traffic</h2>
<p>The complete
	<a href="https://github.com/elastic/examples/blob/master/Security%20Analytics/dns_tunnel_detection/unique_hostnames_watch.json">
	watch file</a> is stored in our
	<a href="https://github.com/elastic/examples/tree/master/Security%20Analytics/dns_tunnel_detection">elastic/examples</a>
	repository along with all of the supporting files shown here. We are going to
walk through the creation of this watch step-by-step.
</p>
<h2>Watch Trigger</h2>
<p>The watch trigger specifies when the execution should start. This watch is
scheduled to execute every 15 minutes.
</p>
<pre class="prettyprint">      &quot;trigger&quot;: {
         &quot;schedule&quot;: {
            &quot;interval&quot;: &quot;15m&quot;
         }
      },
</pre>
<h2>Watch Input</h2>
<p>This first step in creating this watch is to design a set of aggregations to
be used as the input to the watch. We want to find the cardinality of the
hostnames associated with each second-level domain
(e.g. 
	<code>badguy.co.</code>).
</p>
<p>We start with a query that has just two components, a time window and a
whitelist. The time window and whitelist can be customized. Find more on this in
the 
	<a href="#tuning">Tuning the Detector</a> section.
</p>
<p>Next we use a terms aggregation to create buckets for each second-level
domain. Then we apply a sub-aggregation to get the cardinality of the hostnames
within that bucket. Finally we apply a bucket selector aggregation to select
only the buckets having more than 200 unique hostnames. The watch will generate
an alert when the number of unique hostnames breaks this threshold.
</p>
<pre class="prettyprint">GET packetbeat-*/dns/_search
{
  &quot;query&quot;: {
    &quot;bool&quot;: {
      &quot;filter&quot;: {
        &quot;range&quot;: {
          &quot;@timestamp&quot;: {
            &quot;from&quot;: &quot;now-4h&quot;
          }
        }
      },
      &quot;must_not&quot;: {
        &quot;terms&quot;: {
          &quot;dns.question.etld_plus_one&quot;: [
            &quot;akadns.net.&quot;,
            &quot;amazonaws.com.&quot;,
            &quot;apple.com.&quot;,
            &quot;apple-dns.net.&quot;,
            &quot;cloudfront.net.&quot;,
            &quot;icloud.com.&quot;,
            &quot;in-addr.arpa.&quot;,
            &quot;google.com.&quot;,
            &quot;yahoo.com.&quot;
          ]
        }
      }
    }
  },
  &quot;size&quot;: 0,
  &quot;aggs&quot;: {
    &quot;by_domain&quot;: {
      &quot;terms&quot;: {
        &quot;size&quot;: 1000,
        &quot;field&quot;: &quot;dns.question.etld_plus_one&quot;
      },
      &quot;aggs&quot;: {
        &quot;unique_hostnames&quot;: {
          &quot;cardinality&quot;: {
            &quot;field&quot;: &quot;dns.question.name&quot;
          }
        },
        &quot;total_bytes_in&quot;: {
          &quot;sum&quot;: {
            &quot;field&quot;: &quot;bytes_in&quot;
          }
        },
        &quot;total_bytes_out&quot;: {
          &quot;sum&quot;: {
            &quot;field&quot;: &quot;bytes_out&quot;
          }
        },
        &quot;high_num_hostnames&quot;: {
          &quot;bucket_selector&quot;: {
            &quot;buckets_path&quot;: {
              &quot;unique_hostnames&quot;: &quot;unique_hostnames&quot;
            },
            &quot;script&quot;: &quot;params.unique_hostnames &gt; 200&quot;
          }
        }
      }
    }
  }
}
</pre>
<p>The query above relies on the <code>dns.question.etld_plus_one</code> field provided
by Packetbeat 5.x to bucket all requests for a single domain. Packetbeat
creates this field using an embedded copy of the 
	<a href="https://publicsuffix.org/">
	Public Suffix List</a>.
</p>
<h2>Watch Condition</h2>
<p>The watch condition is what determines if an alert is triggered. The
condition here is simple. This says that if any buckets were returned then
trigger the alert.
</p>
<pre class="prettyprint">      &quot;condition&quot;: {
         &quot;script&quot;: {
            &quot;inline&quot;: &quot;ctx.payload.aggregations.by_domain.buckets.size() &gt; 0&quot;
         }
      },
</pre>
<h2>Watch Actions</h2>
<p>The watch actions are executed after the condition is met, and define the
&quot;output&quot; of a watch. For this watch we are sending an email and also writing a
message to the Elasticsearch log. A&#xA0;
	<em>transform</em>&#xA0;script is being
used to manipulate the data so that it renders better in an email.
</p>
<pre class="prettyprint">        &quot;transform&quot;: {
           &quot;script&quot;: {
              &quot;file&quot;: &quot;dns_transform&quot;
           }
        },
        &quot;actions&quot;: {
           &quot;log_domains&quot;: {
              &quot;logging&quot;: {
                 &quot;text&quot;: &quot;The following domain(s) have a high number of unique hostnames: {{ctx.payload.alerts}}&quot;
              }
           },
           &quot;email_alert&quot; : {
            &quot;email&quot;: {
              &quot;to&quot;: &quot;&apos;John Doe &lt;john.doe@example.com&gt;&apos;&quot;,
              &quot;subject&quot;: &quot;Suspected DNS Tunnel Alert&quot;,
              &quot;body&quot;: &quot;The following domain(s) have a high number of unique hostnames: {{ctx.payload.alerts}}&quot;
            }
          }
        }
</pre>
<p>Below is the dns_transform Painless script. It should be placed into the
config/scripts directory of Elasticsearch.
</p>
<pre class="prettyprint">// File: config/scripts/dns_transform.painless
def alerts = ctx.payload.aggregations.by_domain.buckets.stream().collect(Collectors.toMap(p-&gt;p.key,item-&gt;[
        &quot;total_requests&quot; : item.doc_count,
        &quot;unique_hostnames&quot; : item.unique_hostnames.value,
        &quot;total_bytes_in&quot; : item.total_bytes_in.value,
        &quot;total_bytes_out&quot; : item.total_bytes_out.value,
        &quot;total_bytes&quot; : item.total_bytes_in.value + item.total_bytes_out.value
]));
return [&quot;alerts&quot;:alerts];
</pre>
<p>Here is a sample alert. Notice it contains a some additional metrics that can
be used to gauge the severity of the situation.
</p>
<pre class="prettyprint">Date: Fri, 16 Feb 2016 11:00:01 -0500 (EST)
From: Watcher &lt;watcher@example.com&gt;
Message-Id: &lt;201602161600.u0SG01ks024814@example.com&gt;
To: John Doe &lt;john.doe@example.com&gt;
Subject: Suspected DNS Tunnel Alert
The following domain(s) have a high number of unique hostnames:
{badguy.co.={total_requests=222, unique_hostnames=222, total_bytes_in=16716.0,
total_bytes_out=35161.0, total_bytes=51877.0}}
</pre>
<p>When this alert is received, the recipient can take the domain and do a
search using the Discover application in Kibana to find the network clients
responsible for the tunnel.
</p>
<h2>Testing and Results</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2dd8edd48f7693791e/download?uid=blt8aad375573dc7783" data-sys-asset-uid="blt8aad375573dc7783" alt="Unique FQDNs per Second Level Domain">
</p>
<p>This chart shows the top ten domains with the highest number of unique
hostnames over a period of 4 hours. This data was collected from a network with
about 100 devices. During that time window I replayed a network capture
containing a tunnel created by&#xA0;
	<a href="http://code.kryo.se/iodine/">iodine</a>. The tunnel which is operating
under the fictitious domain&#xA0;
	<code>pirate.sea</code>&#xA0;was up for just 20
seconds, and yet it has the highest number of domains.
</p>
<p>If the whitelist from the watch is applied then the tunnel really stands out
among the other domains as seen below.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2d12798d8e381ff0be/download?uid=bltf1e4c3e0ba99f734" data-sys-asset-uid="bltf1e4c3e0ba99f734" alt="Unique FQDNs per Second Level Domain with Whitelist">
</p>
<h2 id="tuning">Tuning the Detector</h2>
<p>There are two variables that can be tuned -- the time window and the unique
hostnames threshold. A smaller time window can be used with a smaller threshold
to make the watch more sensitive to short duration tunnels. In a shorter time
window, domains not being used for tunneling will generally accumulate fewer
unique hostnames.
</p>
<p>Tunnels that move data slowly can be detected using a larger time window. But
using a larger time window means that valid domains with a lot of unique
hostnames, such as CDNs, will cause false positives. So if you use a large time
window you will likely need to add domains to the query&apos;s whitelist.
</p>
<h2>Conclusion</h2>
<p>It was fun combining Packetbeat and Watcher to look for DNS tunnels. Remember
&quot;defense in depth&quot; if you implement a solution like this. It is important to
layer your defenses so that if one layer fails there is another one in place to
detect.
</p></body></html>

tunnel-in-gaza.jpg

brick-tunnel.jpg

Detecting DNS Tunnels with Packetbeat and Watcher

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Software Engineer

Andrew Kroh

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、上記で提供された連絡先に<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">特定の通知</a>が送信されることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシーポリシー</a>をご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p><p>By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.</p>

GDPR Text + Google Cloud

<p>送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。</p><p>このフォームを送信することにより、Elasticが私の連絡先情報をGoogle Cloudに共有することに同意します。また、Elasticが私に食材パッケージを発送する目的で私の情報をベンダーに提供することにも同意します。</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>送信することにより、Elasticの<a href="/jp/legal/serverless-preview-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>送信することにより、<a href="/jp/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud利用規約</a>に同意したものとみなされます。個人情報は<a href="/jp/legal/privacy-statement">Elasticのプライバシーポリシー</a>に基づいて取り扱われます。</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>送信することにより、Elasticの<a href="/jp/legal/terms-of-use" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。</p>

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

このページのコンテンツは、選択された言語ではご利用いただけません。Elasticでは現在、コンテンツをさまざまな言語でご利用いただけるよう取り組んでいます。お使いの言語が準備されるまでお待ちくださるようお願いいたします。

Author

Articles by

Watch now (no PT)

Watch now

See all top stories

すべてのトップストーリーを見る

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Search Integrations

All Solutions

Video for

See when this webinar starts in my time zone

現在のタイムゾーンでこのウェビナー開始時間を見る

Register to Watch

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

他の予定と重なってしまった…という場合も、問題ありません。参加登録されると、録画配信をご利用いただけます。関連コンテンツのご案内もメールでお届けします。