siem-blog-banner.png

siem-blog-thumb.png

Elastic SIEM 7.4.0 released

The new Auditbeat System Module provides four datasets to automatically collect host, process, socket, and user information from hosts.

auditbeat-system-host-dashboard-large.png

auditbeat-system-host-dashboard-small.png

Introducing the Auditbeat System Module

How to run Elasticsearch on Microsoft Azure, deploying with the Azure Marketplace template or manually, and collecting data with Beats and Logstash.

<table style="background: #FFFFD2;">
<tbody>
<tr>
	<td><em>September 3, 2019:</em> Elasticsearch Service on Elastic Cloud is <a href="/blog/elasticsearch-service-on-elastic-cloud-now-available-on-microsoft-azure">now available on Microsoft Azure</a>.
	</td>
</tr>
</tbody>
</table>
<p>I recently had the chance to present at Azure OpenDev, giving an overview of what running and using Elasticsearch and the Elastic Stack looks like today. What I'd like to do today is spend a few minutes going into more detail on some of the topics.<br>
</p>
<p>But first, here's the recording:&nbsp;
</p>
<div style="height: 478px; margin-bottom: 20px;">
	<iframe width="100%" height="478" src="//www.youtube.com/embed/tOqWX9JWEYc?rel=0" class="video-iframe" frameborder="0" allowfullscreen="">
	</iframe>
</div>
<p>See the <a href="https://azure.microsoft.com/en-us/opendev/" target="_blank">event website</a>&nbsp;for the other talks by Microsoft, GitHub, CloudBees / Jenkins, Chef, and HashiCorp.
</p>
<h2>Azure Marketplace</h2>
<p>As demonstrated in the talk, the easiest way to get started with Elastic on Azure is to use the <a href="https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.elasticsearch?tab=Overview" target="_blank">official deployment template in the Azure Marketplace</a>. You can deploy it directly from the Azure Portal and it's going to handle all of the steps required to get Elasticsearch and Kibana up and running: Provisioning instances and storage, deploying and configuring the software, setting up networking and finally bringing everything up.
</p>
<p>We have a <a href="https://www.elastic.co/blog/spinning-up-a-cluster-with-elastics-azure-marketplace-template">good blog post</a>&nbsp;about how to use it.
</p>
<p>Now, there is another way of using it besides running it from the Marketplace. Since it is open source (the sources are <a href="https://github.com/elastic/azure-marketplace">on Github</a>) you can also choose to deploy it from the command line. This allows you to automate things, and to make any customizations to its workings that you might want to do. We have <a href="https://www.elastic.co/blog/elasticsearch-and-kibana-deployments-on-azure">another good blog post</a> about that.
</p>
<p>If you're using the template, please drop us some feedback at <a href="mailto:azuremarketplace@elastic.co">azuremarketplace@elastic.co</a>.
</p>
<h2>Hardware</h2>
<p>Some of the most often asked questions when deploying on Azure are:
</p>
<h5>Which instances should I use?</h5>
<p><i>Keep in mind this section was written in November 2017.</i>
</p>
<p><strong>Elasticsearch - Data Nodes</strong>&nbsp;We found the <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-memory" target="_blank">DS series of memory optimized instances</a> to be a good fit. Like every other data store Elasticsearch is very dependent on the amount of memory available to itself (as the JVM heap) and to the underlying host system (used for the important filesystem cache). You can read a bit more about how memory should be assigned in <a href="https://www.elastic.co/blog/a-heap-of-trouble">this blog post</a>.
</p>
<p>We also recommend using <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/premium-storage" target="_blank">Premium Storage</a>. Backed by Solid State Drives (SSD) it allows Elasticsearch to reach its stored data quickly - and users will benefit from improved response times. <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/managed-disks-overview" target="_blank">Premium Managed Disks</a> also come with encryption at rest (via&nbsp;<a href="https://docs.microsoft.com/en-us/azure/storage/common/storage-service-encryption" target="_blank">Storage Service Encryption</a>).
</p>
<p><strong>Elasticsearch - Master Nodes</strong> For bigger clusters, we recommend having three dedicated master nodes. They will not be storing any data, but will handle cluster management tasks like creating new indices and rebalancing shards. Small D series instances are most often good enough.
</p>
<p><strong>Kibana</strong> Same as master nodes, Kibana has relatively light resource requirements. Most computations are pushed down to Elasticsearch, so you can usually run Kibana on small D series instances as well.
</p>
<p><strong>Logstash</strong> Since it typically does a lot of processing it is best deployed on <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes-compute" target="_blank">the FS series</a>.
</p>
<h2>Availability</h2>
<p>Oftentimes you will want to deploy a highly available Elasticsearch cluster that stays online even in the face of instance or zone failure. Azure has several concepts that help you design redundancy into your deployment and <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/manage-availability" target="_blank">the documentation</a> is a good read.
</p>
<p><strong>Regions</strong> Azure has geographical regions around the world. Each region then contains multiple data centers very close together. You will most likely want to choose whichever region is closest to you, or closest to the users of the system. All nodes of an Elasticsearch cluster should be deployed in the same region.
</p>
<p><strong>Availability Sets</strong> Each tier of the Elastic Stack should be in <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/manage-availability#configure-each-application-tier-into-separate-availability-sets" target="_blank">in its own set</a>. Two instances of Kibana should be in one, two instances of Logstash in another, and the Elasticsearch nodes in a third set.<br>
</p>
<p><strong>Fault and Update Domains</strong> Azure distributes instances across <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/manage-availability#configure-multiple-virtual-machines-in-an-availability-set-for-redundancy" target="_blank">fault and update domains</a>. During a planned maintenance event, only one update domain is going to be rebooted at a time, while only the machines in the same&nbsp;fault domain are sharing&nbsp;a power source and a network switch.&nbsp;Distributing your instances across domains ensures the availability of&nbsp;instances in expected and unexpected circumstances.
</p>
<p><strong>Availablity Zones</strong> Azure is previewing the concept and is supporting <a href="https://docs.microsoft.com/en-us/azure/availability-zones/az-overview" target="_blank">three zones per region</a>. Going forward, this is likely going to be the best way to deploy Elasticsearch. Each zone should have one master-eligible node (or a dedicated master node) and data nodes should be distributed across zones and tagged appropriately using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.0/allocation-awareness.html">Shard Allocation Awareness</a>.
</p>
<p><strong>Scale Sets</strong> When using dedicated&nbsp;Elasticsearch master nodes (see above), using <a href="https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-overview" target="_blank">Scale Sets</a>&nbsp;is a good way to scale the&nbsp;Elasticsearch data nodes&nbsp;up and&nbsp;down as needed.
</p>
<h2>Backup</h2>
<p>Elasticsearch has a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.0/modules-snapshots.html">Snapshot Restore API</a> to ship index files to a remote backup location. An official <a href="https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/repository-azure.html">plugin for Azure</a> is available, and it <a href="https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/repository-azure-usage.html">supports all Standard storage accounts</a>.
</p>
<h2>Collecting Data with Beats and Logstash</h2>
<p><img src="https://api.contentstack.io/v2/uploads/59e3b5551d5a98c32c181792/download?uid=bltee14a95403a23d46" data-sys-asset-uid="bltee14a95403a23d46" alt="Azure reference architecture"><br>
</p>
<h4>Beats</h4>
<p>We've <a href="https://www.elastic.co/blog/brewing-in-beats-enhance-data-with-azure-metadata">recently added support</a> for enriching events collected by the Beats with Azure metadata (instance_id, instance_name, machine_type, region).
</p>
<p>So no matter whether you use <a href="https://www.elastic.co/guide/en/beats/filebeat/6.0/filebeat-overview.html">Filebeat</a>&nbsp;to tail log files, <a href="https://www.elastic.co/guide/en/beats/metricbeat/6.0/metricbeat-overview.html">Metricbeat</a>&nbsp;for system metrics, the new <a href="https://www.elastic.co/guide/en/beats/auditbeat/6.0/auditbeat-overview.html">Auditbeat</a>&nbsp;for Linux audit logs, or any other of the many official and community beats - you will always know on which machine an event originated.
</p>
<h4>Logstash</h4>
<p>In contrast to the Beats which collect data from the source, Logstash is commonly used to receive data from the Beats for further processing - or to pull data from intermediary systems. There are a number of third party input plugins available specifically for Azure:
</p>
<p><strong>Azure Event Hub</strong> Reads data <a href="https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureeventhub" target="_blank">from Event Hub partitions</a>.
</p>
<p><strong>Azure Storage Blobs</strong> Given a storage account name, access key, and container name, it will <a href="https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureblob" target="_blank">read the container contents</a>.
</p>
<p><strong>Azure Service Bus Topics</strong> Reads messages <a href="https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azuretopic" target="_blank">from a Service Bus topic</a>.
</p>
<h2>Summary</h2>
<p>Deploying Elasticsearch and the Elastic Stack on Azure is a great idea, and hopefully this post gives you many pointers on how to do it. Let us know how it goes!
</p>

azure-opendev-large.jpg

azure-opendev-small.jpg

Deploying Elasticsearch on Microsoft Azure

Learn how to get started with the Elastic Stack on Microsoft Azure.

microsoft-azure-clouds-large.png

microsoft-azure-clouds-small.png

Getting Started with the Elastic Stack on Microsoft Azure

Learn how to architect a real-time data pipeline for network packet analysis using Wireshark, Filebeat, Logstash, Ingest Pipelines, Elasticsearch, and Kibana.

network-cables-fullbleed.jpg

network-cables-small.jpg

Analyzing Network Packets with Wireshark, Elasticsearch, and Kibana

How to use the new 5.0 Ingest Node with web logs using Filebeat & Elasticsearch and build awesome dashboards quickly!

<html><head></head><body><p>This is the second part of a two-part series about ingest nodes, a new feature in <a href="https://www.elastic.co/blog/elastic-stack-5-0-0-released">Elasticsearch 5.0</a>.
</p><span id="docs-internal-guid-f273bda2-3fbe-15df-6f33-6a2fa329b55b"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" rel="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">In the <a href="https://www.elastic.co/blog/new-way-to-ingest-part-1">first part</a> we talked about what ingest nodes are, and how to configure and use them. In this second part we will focus on how to use ingest nodes as part of a deployment of the Elastic Stack.
</p><h3 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt;">A quick word about architecture</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Many types of data have to be changed in some way (fields added or removed, text fields parsed, etc.) before it is indexed into Elasticsearch. The tool of choice for this task has so far been Logstash. With the ingest node in Elasticsearch, there is now an alternative for some cases: connecting one of the Beats directly to Elasticsearch and doing the transformations in the ingest node.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p><img src="https://api.contentstack.io/v2/uploads/5820bcf54f97a3ad15b4ed89/download?uid=blt26d27a955d1401fc" data-sys-asset-uid="blt26d27a955d1401fc" alt="Screen Shot 2016-11-07 at 18.41.10.png"></p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">By default, the ingest functionality is enabled on any node. If you know you&#x2019;ll have to process a lot of documents you might want to start a dedicated Elasticsearch node that will only be used for running ingest pipelines. To do that, switch off the master and data node types for that node in <code>elasticsearch.yml</code>&#xA0;(<code>node.master: false</code> and <code>node.data: false</code>) and switch off the ingest node type for all other nodes (<code>node.ingest: false</code>).<br></div><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">When consuming data from either <a href="https://www.elastic.co/products/beats/filebeat">Filebeat</a> or <a href="https://www.elastic.co/products/beats/winlogbeat">Winlogbeat</a> it might also no longer be necessary to use a message queue. Both of these Beats (but not others at this time) can handle backpressure: If Elasticsearch can&#x2019;t keep up with indexing requests due to a large burst of events, these Beats will fall behind in sending the most recent data but will maintain a pointer to the last successfully indexed position in the files (Filebeat) or the event log (Winlogbeat). Once Elasticsearch has again caught up with the indexing load, the Beats will&#xA0;catch up to the most recent events.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">It might still be a good idea to use a message queue such as Kafka or Redis, for example to decouple Beats and Elasticsearch architecturally or to be able to take Elasticsearch offline, e.g. for an upgrade. Keep in mind though that the ingest node is a push-based system, it will not read from a message queue. So when using a queue, it&#x2019;s necessary to use Logstash to push the data from the queue into Elasticsearch and its ingest node.
</p><h3 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt;">Configure Filebeat and Elasticsearch</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">To deploy the new architecture described above we need to configure three components: Filebeat, Elasticsearch (incl. the ingest node) and Kibana. We&#x2019;ll continue to use the example of ingesting web access logs from an Apache httpd web server.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">We&#x2019;ll take the configuration of each component&#xA0;in turn:
</p><h4 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Elasticsearch</h4><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">All the configuration was already described in Part 1 of this blog post series, but for reference we&#x2019;ll include the relevant pieces here as well.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">We&#x2019;ll be using two ingest plugins that need to be installed first:
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><code>bin/elasticsearch-plugin install ingest-geoip</code>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><code>bin/elasticsearch-plugin install ingest-user-agent</code>
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Having done that successfully, we can create an ingest pipeline for web access logs (the format is that used by Kibana&#x2019;s Console):
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><pre class="prettyprint">PUT _ingest/pipeline/access_log
{
  &quot;description&quot; : &quot;Ingest pipeline for Combined Log Format&quot;,
  &quot;processors&quot; : [
    {
      &quot;grok&quot;: {
        &quot;field&quot;: &quot;message&quot;,
        &quot;patterns&quot;: [&quot;%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \&quot;%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\&quot; %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}&quot;]
      }
    },
    {
      &quot;date&quot;: {
        &quot;field&quot;: &quot;timestamp&quot;,
        &quot;formats&quot;: [ &quot;dd/MMM/YYYY:HH:mm:ss Z&quot; ]
      }
    },
    {
      &quot;geoip&quot;: {
        &quot;field&quot;: &quot;clientip&quot;
      }
    },
    {
      &quot;user_agent&quot;: {
        &quot;field&quot;: &quot;agent&quot;
      }
    }
  ]
}
</pre><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Note that the order of processors is important, as they are executed one after the other for each incoming document. First we use <code>grok</code> to extract the fields (<code>clientip</code>, <code>timestamp</code>, <code>bytes</code>, etc.), and then we use the appropriate processors (<code>date</code>, <code>geoip</code>, <code>user_agent</code>) on the extracted data.
</p><h4 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Filebeat</h4><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Filebeat is used to collect the log messages line by line from the log file. In the&#xA0;<code>filebeat.yml</code>&#xA0;configuration file this will&#xA0;look something like this:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/apache2/access_log
</pre><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Further down, we can&#xA0;configure the Elasticsearch output:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">output.elasticsearch:
  hosts: [&lt;insert-your-es-host&gt;:&lt;es-port&gt;]
  template.name: &quot;access_log&quot;
  template.path: &lt;insert-path-to-template&gt;
  pipeline: access_log
</pre><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><code>template.path</code> has to point to a file containing an Elasticsearch template. Filebeat ships with a file called <code>filebeat.template.json</code> that you can modify. For the example in this blog post, we need to&#xA0;add one field to the <code>properties</code> section:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">&quot;geoip&quot;: {
  &quot;properties&quot;: {
    &quot;location&quot;: {
      &quot;type&quot;: &quot;geo_point&quot;
    }
  }
}
</pre><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">This will allow us to visualise the locations of IP addresses in our logs on a map in Kibana later.
</p><h3 dir="ltr">Ingest the data</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">We&#x2019;re good to go! We can run Filebeat to start indexing logs into our Elasticsearch:
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><code>./filebeat -c filebeat.yml -e</code>
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Note: <code>-e</code> causes Filebeat to log to stderr instead of a log file and shouldn&#x2019;t be used in production.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Using the <code>_cat</code> API we can check that the data has been successfully ingested:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">GET _cat/indices/filebeat*?v

health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-2016.10.28 W_p-RZwQRWWn_S49jwaTFw   5   1     300000            0    212.7mb        212.7mb
</pre><h3 dir="ltr">Use it in Kibana</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">To use the data in Kibana, we&apos;ll configure an index pattern for <code>filebeat-*</code>. These configurations are at&#xA0;<code>Management -&gt; Index Pattern</code>.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><img src="https://lh3.googleusercontent.com/MUA-DpQhzKPB1rvH1_F3T1lYeigsE1Bpnn1JwLps85MEFCR-6Niw3SpQRBBNZl51WVXb91HskSWgE7B_7SkUr_Zs7aI5d2yOukPNXTIt9AoGquwhsVBWuUQ108Ev3D8UrIyhDRm6" alt="Screen Shot 2016-10-28 at 17.05.29.png" width="624" height="391">
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">To explore the data a bit, we can use Kibana&#x2019;s Discover view. The search bar allows us to explore website visits by country, requested resource and other criteria.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">For example, in this dataset we&#xA0;can search for visits to the blog section of the website from Amsterdam using the query <code>request:blog AND geoip.city_name:&quot;Amsterdam&quot;</code>:
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><img src="https://lh3.googleusercontent.com/oc_GoxYz-voMcZskz7KsRSmyLt7HpzS7eZmtdbYz9K5hqHBrAhcs6yzOHameuCUFWC0H_YPjZLnEOMSLkKew8kfOT5s2OFz7tNxop1p4MRUV-81Sy-vDSpEiajHALy2RSwbuQGkl" alt="Screen Shot 2016-10-28 at 17.23.23.png" width="624" height="391">
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">In the Visualize section, we can build a pie chart of Top Countries:
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><img src="https://lh5.googleusercontent.com/iw28NzWTqzC3-qujyg97mXdvvGHwXepEDLAILHWAlEIEY3wSwo_hqHGVeQ_k3MYF3JG7T_iH5m-k6gYD4dTLCIyWyoR0swz-VS08fYZVBQODfUdZGrDivb3q7QKZ8kn66ADEcjWD" alt="Screen Shot 2016-10-28 at 17.32.12.png" width="624" height="391">
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">We can build complex visualisations in Timelion, e.g. with this query we see the fluctuations in the amount of requests from the North America and Europe overlaid with the number of all requests.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><code>.es().bars(10).label(World) .es(q=&quot;geoip.continent_name:North America&quot;, metric=count).points(10, fill=10).label(&quot;North America&quot;) .es(q=&quot;geoip.continent_name:Europe&quot;, metric=count).points(10, fill=10).label(Europe)</code>
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><img src="https://lh3.googleusercontent.com/lG9Zm-EBMw3sbYIHxl4CykBWVYVGKUyQfnC7GBR7HosFhqwKzSq0wBqUrptJWL6dAWm6uReYM9ADx5l1ciB6T79OGC5h1wtnyxIHVOcRGJpC2SvtagYFV4mnUNNJ3wTD3pRnpiyF" alt="Screen Shot 2016-10-28 at 17.40.52.png" width="624" height="391">
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">A complete dashboard for web traffic might look something like this:
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><img src="https://lh6.googleusercontent.com/KtM9vhLnX1yhhrfy4pHHQYgC7vvSkcY6vMXxZgbFhmQyPTh4EMci8eSqOy1u46pc2R27vqW_VCTCx7dhp9oyI2ws5BdPKSji1nY1ZLBPTrNPc5LQVe80gstMU4RO3IyMRtvlWoZn" alt="Screen Shot 2016-11-01 at 01.30.13.png" width="624" height="391">
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><h3 dir="ltr">Conclusion</h3><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" rel="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">With the ingest node, there is now a way to transform data inside Elasticsearch&#xA0;before indexing it. This is especially useful if only&#xA0;simpler operations are required, while&#xA0;more complex ones can still be performed using Logstash. Written in Java, operations performed in the ingest node are very efficient.
</p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">When you don&#x2019;t need the additional power and flexibility of Logstash filters,&#xA0;this allows you to simplify your architecture for simpler use cases. And with Kibana and Timelion as one of its new built-in features you have the perfect tool for visualising the data.</p><p><br>
</p></span></body></html>

elasticsearch-ingest-tunnel-color-speed-2000.jpg

elasticsearch-ingest-tunnel-color-speed-720.jpeg

A New Way To Ingest - Part 2

Ingest Nodes are a new feature in Elasticsearch 5 that allows you to change data right before it is indexed, e.g. extracting fields from long message strings.

<html><head></head><body><p>With <a href="https://www.elastic.co/v5">the upcoming release of version 5.0 of the Elastic Stack</a>, it is time we took a closer look at how to use one of the new features, Ingest Nodes.
</p><span id="docs-internal-guid-186bcb51-4c6e-d0f1-1f66-cc82c04a51ad"><h3 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt;" rel="line-height:1.38;margin-top:20pt;margin-bottom:6pt;">What are Ingest Nodes?</h3><p>Ingest Nodes are a new type of Elasticsearch node you can use to perform common data transformation and enrichments.
</p><p>Each task is represented by a processor. Processors are configured to form pipelines.
</p><p>At the time of writing the Ingest Node had 20 built-in processors, for example grok, date, gsub, lowercase/uppercase, remove and rename. You can find a full list&#xA0;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-processors.html">in the documentation</a>.<br>
</p><p>Besides those, there are currently also three Ingest plugins:<br>
</p><ol>
	<li dir="ltr"><a href="https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-attachment.html">Ingest Attachment</a>&#xA0;converts binary documents like Powerpoints, Excel Spreadsheets, and PDF documents to text and metadata</li>
	<li dir="ltr"><a href="https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-geoip.html">Ingest Geoip</a>&#xA0;looks up the geographic locations of IP addresses in an internal database </li>
	<li dir="ltr"><a href="https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-user-agent.html">Ingest user agent</a>&#xA0;parses and extracts information from the user agent strings used by browsers and other applications when using HTTP</li>
</ol><h3 dir="ltr">Create and Use an Ingest Pipeline</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">You configure a new ingest pipeline with the <code>_ingest</code> API endpoint.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><div style="padding: 3%; background-color: rgb(180, 180, 180);">
	<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color: rgb(255, 255, 255);">Note: Throughout this blog post, when showing requests to Elasticsearch we are using the format of <a href="https://www.elastic.co/guide/en/kibana/5.0/console-kibana.html">Console</a>.</span>
	</p>
</div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="background-color: rgb(247, 150, 70);"><span style="color: rgb(255, 255, 255);"><br></span></span>
</p><pre class="prettyprint">PUT _ingest/pipeline/rename_hostname
{
  &quot;processors&quot;: [
    {
      &quot;rename&quot;: {
        &quot;field&quot;: &quot;hostname&quot;,
        &quot;target_field&quot;: &quot;host&quot;,
        &quot;ignore_missing&quot;: true
      }
    }
  ]
}
</pre><p>In this example, we configure a pipeline called <code>rename_hostname</code> that simply takes the field <code>hostname</code> and renames it to <code>host</code>. &#xA0;If the <code>hostname</code> field does not exist, the processor continues without error.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">To use this pipeline, there&#x2019;s several ways.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">When using plain Elasticsearch APIs, you specify the <code>pipeline</code> parameter in the query string, e.g.:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">POST server/values/?pipeline=rename_hostname
{
  &quot;hostname&quot;: &quot;myserver&quot;
}
</pre><p>In Logstash, you add the <code>pipeline</code> parameter to the elasticsearch output:
</p><pre class="prettyprint">output {
  elasticsearch {
    hosts =&gt; &quot;192.168.100.39&quot;
    index =&gt; &quot;server&quot;
    pipeline =&gt; &quot;rename_hostname&quot;
  }
}
</pre><p>Similarly, you add a parameter to the elasticsearch output of any Beat:<br>
</p><p><span></span>
</p><pre class="prettyprint">output.elasticsearch:
  hosts: [&quot;192.168.100.39:9200&quot;]
  index: &quot;server&quot;
  pipeline: &quot;convert_value&quot;
</pre><div style="padding: 3%; background-color: rgb(180, 180, 180);">
	<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color: rgb(255, 255, 255);">Note: In alpha versions of 5.0, you had to use <code>parameters.pipeline</code> in the Beats configuration.</span>
	</p>
</div><h3>Simulate</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">When configuring a new pipeline, it is often very valuable to be able to test it before feeding it with real data - and only then discovering that it throws an error!
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">For that, there is the&#xA0;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html">Simulate API</a>:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">POST _ingest/pipeline/rename_hostname/_simulate
{
  &quot;docs&quot;: [
    {
      &quot;_source&quot;: {
        &quot;hostname&quot;: &quot;myserver&quot;
      }
    }
  ]
}
</pre><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">The result shows us that our field has been successfully renamed:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">       [...]
        &quot;_source&quot;: {
          &quot;host&quot;: &quot;myserver&quot;
        },
        [...]
</pre><h3 dir="ltr">A real-world example: Web Logs
</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Let&#x2019;s turn to something from the real world: Web logs.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">This is an example of an access log in the&#xA0;<a href="http://httpd.apache.org/docs/current/mod/mod_log_config.html">Combined Log Format</a> supported by both Apache httpd and nginx:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre style="white-space: normal">212.87.37.154 - - [12/Sep/2016:16:21:15 +0000] &quot;GET /favicon.ico HTTP/1.1&quot; 200 3638 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36&quot;
</pre><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">As you can see, it contains several pieces of information: IP address, timestamp, a user agent string, and so on.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">To allow fast search and visualisation we need to give every piece its own field in Elasticsearch. It would also be useful to know where this request is coming from. We can do all this with the following Ingest pipeline.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">PUT _ingest/pipeline/access_log
{
  &quot;description&quot; : &quot;Ingest pipeline for Combined Log Format&quot;,
  &quot;processors&quot; : [
    {
      &quot;grok&quot;: {
        &quot;field&quot;: &quot;message&quot;,
        &quot;patterns&quot;: [&quot;%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \&quot;%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\&quot; %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}&quot;]
      }
    },
    {
      &quot;date&quot;: {
        &quot;field&quot;: &quot;timestamp&quot;,
        &quot;formats&quot;: [ &quot;dd/MMM/YYYY:HH:mm:ss Z&quot; ]
      }
    },
    {
      &quot;geoip&quot;: {
        &quot;field&quot;: &quot;clientip&quot;
      }
    },
    {
      &quot;user_agent&quot;: {
        &quot;field&quot;: &quot;agent&quot;
      }
    }
  ]
}
</pre><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">It contains a total of four processors:
</p><ol>
	<li dir="ltr"><code>grok</code> uses a regular expression to parse the whole log line into individual fields.</li>
	<li dir="ltr"><code>date</code> identifies the timestamp of the document.</li>
	<li dir="ltr"><code>geoip</code> takes the IP address of the requester and looks it up in an internal database to determine its geographical location.</li>
	<li dir="ltr"><code>user_agent</code> takes the user agent string and splits it up into individual components.</li>
</ol><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Since the last two processors are plugins that do not ship with Elasticsearch by default we will have to install them first:
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent
</pre><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">To test our pipeline, we can again use the Simulate API (the double quotes inside <code>message</code> have to be escaped):
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><pre class="prettyprint">POST _ingest/pipeline/access_log/_simulate
{
  &quot;docs&quot;: [
    {
      &quot;_source&quot;: {
        &quot;message&quot;: &quot;212.87.37.154 - - [12/Sep/2016:16:21:15 +0000] \&quot;GET /favicon.ico HTTP/1.1\&quot; 200 3638 \&quot;-\&quot; \&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36\&quot;&quot;
      }
    }
  ]
}
</pre><p>The result from Elasticsearch shows us that this worked:<br>
</p><pre class="prettyprint">{
  &quot;docs&quot;: [
    {
      &quot;doc&quot;: {
        &quot;_index&quot;: &quot;_index&quot;,
        &quot;_type&quot;: &quot;_type&quot;,
        &quot;_id&quot;: &quot;_id&quot;,
        &quot;_source&quot;: {
          &quot;request&quot;: &quot;/favicon.ico&quot;,
          &quot;agent&quot;: &quot;\&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36\&quot;&quot;,
          &quot;geoip&quot;: {
            &quot;continent_name&quot;: &quot;Europe&quot;,
            &quot;city_name&quot;: null,
            &quot;country_iso_code&quot;: &quot;DE&quot;,
            &quot;region_name&quot;: null,
            &quot;location&quot;: {
              &quot;lon&quot;: 9,
              &quot;lat&quot;: 51
            }
          },
          &quot;auth&quot;: &quot;-&quot;,
          &quot;ident&quot;: &quot;-&quot;,
          &quot;verb&quot;: &quot;GET&quot;,
          &quot;message&quot;: &quot;212.87.37.154 - - [12/Sep/2016:16:21:15 +0000] \&quot;GET /favicon.ico HTTP/1.1\&quot; 200 3638 \&quot;-\&quot; \&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36\&quot;&quot;,
          &quot;referrer&quot;: &quot;\&quot;-\&quot;&quot;,
          &quot;@timestamp&quot;: &quot;2016-09-12T16:21:15.000Z&quot;,
          &quot;response&quot;: 200,
          &quot;bytes&quot;: 3638,
          &quot;clientip&quot;: &quot;212.87.37.154&quot;,
          &quot;httpversion&quot;: &quot;1.1&quot;,
          &quot;user_agent&quot;: {
            &quot;patch&quot;: &quot;2743&quot;,
            &quot;major&quot;: &quot;52&quot;,
            &quot;minor&quot;: &quot;0&quot;,
            &quot;os&quot;: &quot;Mac OS X 10.11.6&quot;,
            &quot;os_minor&quot;: &quot;11&quot;,
            &quot;os_major&quot;: &quot;10&quot;,
            &quot;name&quot;: &quot;Chrome&quot;,
            &quot;os_name&quot;: &quot;Mac OS X&quot;,
            &quot;device&quot;: &quot;Other&quot;
          },
          &quot;timestamp&quot;: &quot;12/Sep/2016:16:21:15 +0000&quot;
        },
        &quot;_ingest&quot;: {
          &quot;timestamp&quot;: &quot;2016-09-13T14:35:58.746+0000&quot;
        }
      }
    }
  ]
}
</pre><h3>Next</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">In the second part we will show how to set up an ingestion pipeline using Filebeat, Elasticsearch and Kibana to ingest and visualize web logs.
</p></span></body></html>

Articles by Christoph Wurm

Elastic SIEM 7.4.0 released

Introducing the Auditbeat System Module

Deploying Elasticsearch on Microsoft Azure

Getting Started with the Elastic Stack on Microsoft Azure

Analyzing Network Packets with Wireshark, Elasticsearch, and Kibana

A New Way To Ingest - Part 2

A New Way To Ingest - Part 1

Products & Solutions

Company

Resources

Featured topics

News

Articles by Christoph Wurm

Elastic SIEM 7.4.0 released

Introducing the Auditbeat System Module

Deploying Elasticsearch on Microsoft Azure

Getting Started with the Elastic Stack on Microsoft Azure

Analyzing Network Packets with Wireshark, Elasticsearch, and Kibana

A New Way To Ingest - Part 2

A New Way To Ingest - Part 1