We’d like to announce the first release candidate of the Elastic Stack 8.0.0 and the continuation of the Pioneer Program. Give it a try and please share all the bugs you find on GitHub.

genericrelease.png

Elastic Stack 8.0.0-rc1 released

audit-letters-full-bleed.jpg

audit-clipboard-cc.jpg

Introducing Auditbeat: Ship Linux Audit Logs to Elasticsearch and More

container-loading-small.jpg

container-ship-rena-small.jpg

Monitoring Container Resource Usage with Metricbeat

How to use the Winlogbeat and Kibana to visualize logon events from Windows event logs.

winlogbeat-rdp-map-full-bleed.jpg

winlogbeat-account-usage-dashboard.jpg

Monitoring Windows Logons with Winlogbeat

<html><head></head><body><p><strong>Updated January 12, 2017:</strong> This post was updated to reflect changes in Packetbeat 5.x and Elasticsearch 5.x.
</p>


<p>Data observed from monitoring DNS traffic on a network can be used as an
indicator of compromise (IOC). This blog post will discuss how Elasticsearch and
Watcher can be used with Packetbeat to alert when possible malware activity is
detected.
</p>
<p><a href="/products/beats/packetbeat"><em>Packetbeat</em></a> is our open
source packet analyzer. It monitors the traffic on your network and indexes the
DNS requests and responses into Elasticsearch where aggregations can be used to
help make sense of the data.
</p>
<p><a href="/products/x-pack/alerting"><em>Alerting</em></a> (formerly <em>Watcher</em>)
is part of 
	<a href="/products/x-pack">X-Pack</a> and it provides alerting and
notifications based on changes in your data.
</p>
<p>There are many use cases for alerting on data collected by Packetbeat such as
alerting when the response times for web requests are above a threshold or when
there is spike in HTTP errors returned by your web servers. The alerting
described in this article has applications in network security. We are going to
look at one specific use case -- detecting data exfiltration over DNS tunnels.
</p>
<h2>Detecting DNS Tunnels</h2>
<p>Tunnels can be established over the DNS protocol to covertly move data or
provide a command and control channel for malware. Often this technique is used
to bypass the protections of corporate firewalls and proxy servers. Tunneling
works by encoding data in DNS requests and responses. The client issues a query
for a hostname and that query is eventually forwarded to the authoritative name
server associated with the domain.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2d58208ba076e29b7d/download?uid=bltf505dadaa68339b8" data-sys-asset-uid="bltf505dadaa68339b8" alt="Packetbeat Deployment Architecture">
</p>
<p>There are a lot of different techniques that can be employed for detecting
such traffic. We are going to look at using the number of unique hostnames for a
domain as an IOC. DNS tunneling utilities must use a new hostname for each
request which leads to a much higher number of hostnames present for the
malicious domains in comparison to legitimate domains.
</p>
<h2>Packetbeat Setup</h2>
<p>The first step is to install Packetbeat and configure it to collect DNS
traffic. For this setup, the server running Packetbeat is connected to a port
mirror so that Packetbeat can observe all the traffic between the local network
and the Internet. The Packetbeat documentation has a great 
	<a href="/guide/en/beats/packetbeat/current/packetbeat-getting-started.html">Getting
Started guide
	</a> that explains installation and setup procedure, so I will just
show the configuration used.
</p>
<pre class="prettyprint"># /etc/packetbeat/packetbeat.yml
packetbeat.interfaces.device: en0
packetbeat.protocols.dns:
  ports: [53]
  include_authorities: true
  include_additionals: true
output.elasticsearch:
  hosts: [&quot;localhost:9200&quot;]
</pre>
<h2>Watch your DNS Traffic</h2>
<p>The complete
	<a href="https://github.com/elastic/examples/blob/master/Security%20Analytics/dns_tunnel_detection/unique_hostnames_watch.json">
	watch file</a> is stored in our
	<a href="https://github.com/elastic/examples/tree/master/Security%20Analytics/dns_tunnel_detection">elastic/examples</a>
	repository along with all of the supporting files shown here. We are going to
walk through the creation of this watch step-by-step.
</p>
<h2>Watch Trigger</h2>
<p>The watch trigger specifies when the execution should start. This watch is
scheduled to execute every 15 minutes.
</p>
<pre class="prettyprint">      &quot;trigger&quot;: {
         &quot;schedule&quot;: {
            &quot;interval&quot;: &quot;15m&quot;
         }
      },
</pre>
<h2>Watch Input</h2>
<p>This first step in creating this watch is to design a set of aggregations to
be used as the input to the watch. We want to find the cardinality of the
hostnames associated with each second-level domain
(e.g. 
	<code>badguy.co.</code>).
</p>
<p>We start with a query that has just two components, a time window and a
whitelist. The time window and whitelist can be customized. Find more on this in
the 
	<a href="#tuning">Tuning the Detector</a> section.
</p>
<p>Next we use a terms aggregation to create buckets for each second-level
domain. Then we apply a sub-aggregation to get the cardinality of the hostnames
within that bucket. Finally we apply a bucket selector aggregation to select
only the buckets having more than 200 unique hostnames. The watch will generate
an alert when the number of unique hostnames breaks this threshold.
</p>
<pre class="prettyprint">GET packetbeat-*/dns/_search
{
  &quot;query&quot;: {
    &quot;bool&quot;: {
      &quot;filter&quot;: {
        &quot;range&quot;: {
          &quot;@timestamp&quot;: {
            &quot;from&quot;: &quot;now-4h&quot;
          }
        }
      },
      &quot;must_not&quot;: {
        &quot;terms&quot;: {
          &quot;dns.question.etld_plus_one&quot;: [
            &quot;akadns.net.&quot;,
            &quot;amazonaws.com.&quot;,
            &quot;apple.com.&quot;,
            &quot;apple-dns.net.&quot;,
            &quot;cloudfront.net.&quot;,
            &quot;icloud.com.&quot;,
            &quot;in-addr.arpa.&quot;,
            &quot;google.com.&quot;,
            &quot;yahoo.com.&quot;
          ]
        }
      }
    }
  },
  &quot;size&quot;: 0,
  &quot;aggs&quot;: {
    &quot;by_domain&quot;: {
      &quot;terms&quot;: {
        &quot;size&quot;: 1000,
        &quot;field&quot;: &quot;dns.question.etld_plus_one&quot;
      },
      &quot;aggs&quot;: {
        &quot;unique_hostnames&quot;: {
          &quot;cardinality&quot;: {
            &quot;field&quot;: &quot;dns.question.name&quot;
          }
        },
        &quot;total_bytes_in&quot;: {
          &quot;sum&quot;: {
            &quot;field&quot;: &quot;bytes_in&quot;
          }
        },
        &quot;total_bytes_out&quot;: {
          &quot;sum&quot;: {
            &quot;field&quot;: &quot;bytes_out&quot;
          }
        },
        &quot;high_num_hostnames&quot;: {
          &quot;bucket_selector&quot;: {
            &quot;buckets_path&quot;: {
              &quot;unique_hostnames&quot;: &quot;unique_hostnames&quot;
            },
            &quot;script&quot;: &quot;params.unique_hostnames &gt; 200&quot;
          }
        }
      }
    }
  }
}
</pre>
<p>The query above relies on the <code>dns.question.etld_plus_one</code> field provided
by Packetbeat 5.x to bucket all requests for a single domain. Packetbeat
creates this field using an embedded copy of the 
	<a href="https://publicsuffix.org/">
	Public Suffix List</a>.
</p>
<h2>Watch Condition</h2>
<p>The watch condition is what determines if an alert is triggered. The
condition here is simple. This says that if any buckets were returned then
trigger the alert.
</p>
<pre class="prettyprint">      &quot;condition&quot;: {
         &quot;script&quot;: {
            &quot;inline&quot;: &quot;ctx.payload.aggregations.by_domain.buckets.size() &gt; 0&quot;
         }
      },
</pre>
<h2>Watch Actions</h2>
<p>The watch actions are executed after the condition is met, and define the
&quot;output&quot; of a watch. For this watch we are sending an email and also writing a
message to the Elasticsearch log. A&#xA0;
	<em>transform</em>&#xA0;script is being
used to manipulate the data so that it renders better in an email.
</p>
<pre class="prettyprint">        &quot;transform&quot;: {
           &quot;script&quot;: {
              &quot;file&quot;: &quot;dns_transform&quot;
           }
        },
        &quot;actions&quot;: {
           &quot;log_domains&quot;: {
              &quot;logging&quot;: {
                 &quot;text&quot;: &quot;The following domain(s) have a high number of unique hostnames: {{ctx.payload.alerts}}&quot;
              }
           },
           &quot;email_alert&quot; : {
            &quot;email&quot;: {
              &quot;to&quot;: &quot;&apos;John Doe &lt;john.doe@example.com&gt;&apos;&quot;,
              &quot;subject&quot;: &quot;Suspected DNS Tunnel Alert&quot;,
              &quot;body&quot;: &quot;The following domain(s) have a high number of unique hostnames: {{ctx.payload.alerts}}&quot;
            }
          }
        }
</pre>
<p>Below is the dns_transform Painless script. It should be placed into the
config/scripts directory of Elasticsearch.
</p>
<pre class="prettyprint">// File: config/scripts/dns_transform.painless
def alerts = ctx.payload.aggregations.by_domain.buckets.stream().collect(Collectors.toMap(p-&gt;p.key,item-&gt;[
        &quot;total_requests&quot; : item.doc_count,
        &quot;unique_hostnames&quot; : item.unique_hostnames.value,
        &quot;total_bytes_in&quot; : item.total_bytes_in.value,
        &quot;total_bytes_out&quot; : item.total_bytes_out.value,
        &quot;total_bytes&quot; : item.total_bytes_in.value + item.total_bytes_out.value
]));
return [&quot;alerts&quot;:alerts];
</pre>
<p>Here is a sample alert. Notice it contains a some additional metrics that can
be used to gauge the severity of the situation.
</p>
<pre class="prettyprint">Date: Fri, 16 Feb 2016 11:00:01 -0500 (EST)
From: Watcher &lt;watcher@example.com&gt;
Message-Id: &lt;201602161600.u0SG01ks024814@example.com&gt;
To: John Doe &lt;john.doe@example.com&gt;
Subject: Suspected DNS Tunnel Alert
The following domain(s) have a high number of unique hostnames:
{badguy.co.={total_requests=222, unique_hostnames=222, total_bytes_in=16716.0,
total_bytes_out=35161.0, total_bytes=51877.0}}
</pre>
<p>When this alert is received, the recipient can take the domain and do a
search using the Discover application in Kibana to find the network clients
responsible for the tunnel.
</p>
<h2>Testing and Results</h2>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2dd8edd48f7693791e/download?uid=blt8aad375573dc7783" data-sys-asset-uid="blt8aad375573dc7783" alt="Unique FQDNs per Second Level Domain">
</p>
<p>This chart shows the top ten domains with the highest number of unique
hostnames over a period of 4 hours. This data was collected from a network with
about 100 devices. During that time window I replayed a network capture
containing a tunnel created by&#xA0;
	<a href="http://code.kryo.se/iodine/">iodine</a>. The tunnel which is operating
under the fictitious domain&#xA0;
	<code>pirate.sea</code>&#xA0;was up for just 20
seconds, and yet it has the highest number of domains.
</p>
<p>If the whitelist from the watch is applied then the tunnel really stands out
among the other domains as seen below.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d2d12798d8e381ff0be/download?uid=bltf1e4c3e0ba99f734" data-sys-asset-uid="bltf1e4c3e0ba99f734" alt="Unique FQDNs per Second Level Domain with Whitelist">
</p>
<h2 id="tuning">Tuning the Detector</h2>
<p>There are two variables that can be tuned -- the time window and the unique
hostnames threshold. A smaller time window can be used with a smaller threshold
to make the watch more sensitive to short duration tunnels. In a shorter time
window, domains not being used for tunneling will generally accumulate fewer
unique hostnames.
</p>
<p>Tunnels that move data slowly can be detected using a larger time window. But
using a larger time window means that valid domains with a lot of unique
hostnames, such as CDNs, will cause false positives. So if you use a large time
window you will likely need to add domains to the query&apos;s whitelist.
</p>
<h2>Conclusion</h2>
<p>It was fun combining Packetbeat and Watcher to look for DNS tunnels. Remember
&quot;defense in depth&quot; if you implement a solution like this. It is important to
layer your defenses so that if one layer fails there is another one in place to
detect.
</p></body></html>

tunnel-in-gaza.jpg

brick-tunnel.jpg

Detecting DNS Tunnels with Packetbeat and Watcher

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Software Engineer

Andrew Kroh

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/elastic-cloud-account-terms" target="_self">Conditions générales d'utilisation</a>, et vous acceptez de recevoir les <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">types de notification sélectionnés</a> à l'adresse que vous avez fournie plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p><p>By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.</p>

GDPR Text + Google Cloud

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/serverless-preview-terms" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>En envoyant votre demande d'inscription, vous acceptez les <a href="/fr/agreements/global/cloud-services-monthly" rel="nofollow">conditions générales d'utilisation d'Elastic Cloud</a>. Vos données personnelles seront traitées conformément à la <a href="/fr/legal/privacy-statement">déclaration de confidentialité d'Elastic</a>.
</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/terms-of-use" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.</p>

Explore similar demos

Overview

You'll also receive an email with related content.

You'll also receive an email with related content

Nous vous enverrons plus de contenu susceptible de vous intéresser.

Speakers

Close

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Le contenu de cette page n'est pas disponible dans la langue sélectionnée. Chez Elastic, nous mettons tout en œuvre pour vous proposer du contenu dans différentes langues. En attendant, nous vous remercions de votre patience !

See more insights

Articles by

Author

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Voir à quelle heure débute le webinar dans mon fuseau horaire

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Si vous ne pouvez pas vous libérer le jour J, inscrivez-vous et nous vous enverrons l'enregistrement. Vous recevrez aussi un e-mail contenant des ressources connexes.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

Thank you for registering. We will send you a confirmation email soon.

Nous vous remercions de votre inscription. Vous allez recevoir un e-mail de confirmation.

Author

Articles by Andrew Kroh

Software Engineer, Elastic

Elastic Stack 8.0.0-rc1 released

Introducing Auditbeat: Ship Linux Audit Logs to Elasticsearch and More

Monitoring Container Resource Usage with Metricbeat

Monitoring Windows Logons with Winlogbeat

Detecting DNS Tunnels with Packetbeat and Watcher