<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><h2>Using Logstash and Kibana on Found by Elastic</h2><p>This is part one of a two post blog series, aiming to demonstrate how to feed logs from IIS into Elasticsearch and Kibana via Logstash, using the hosted services provided by Found by Elastic. This post will deal with setting up the basic functionality and securing connections. <a href="/blog/found-analyzing-weblogs-with-elasticsearch-part-2">Part 2</a> will show how to configure Logstash to read from IIS log files, and how to use Kibana 4 to visualize web traffic.
</p><section id="getting-the-bits" class="level2">
<h2><a href="#getting-the-bits">Getting the Bits</a></h2>
<p>For this demo I will be running Logstash and Kibana from my Windows laptop. If you want to follow along, download and extract Logstash 1.5.RC4 or later, and Kibana 4.0.2 or later from <a href="/downloads">https://www.elastic.co/downloads</a>. We need Logstash 1.5, since it introduces support for SSL connections.
</p>
</section><section id="creating-an-elasticsearch-cluster" class="level2">
<h2><a href="#creating-an-elasticsearch-cluster">Creating an Elasticsearch Cluster</a></h2>
<p>Creating a new trial cluster in Found is just a matter of logging in and pressing a button. It takes a few seconds until the cluster is ready, and a screen with some basic information on how to connect pops up. We need the address for the HTTPS endpoint, so copy that out.
</p>
</section><section id="configuring-logstash" class="level2">
<h2><a href="#configuring-logstash">Configuring Logstash</a></h2>
<p>Now, with the brand new SSL connection option in Logstash, connecting to Found is as simple as this Logstash configuration:
</p>
<pre class="prettyprint">input { stdin{} }
output {
  elasticsearch {
    protocol =&gt; http
    host =&gt; REPLACE_WITH_FOUND_CLUSTER_HOSTNAME (full uri in quotes)
    port =&gt; &quot;9243&quot; # Check the port also
    ssl =&gt; true
}
stdout { codec =&gt; rubydebug }
}
</pre>
<p>Save the file as found.conf
</p>
<p>Start up Logstash using
</p>
<p><code>bin\logstash.bat agent --verbose -f found.conf</code>
</p>
<p>You should see a message similar to
</p>
<pre class="prettyprint">Create client to elasticsearch server on `https://....foundcluster.com:9243`: {:level=&gt;:info}
</pre>
<p>Once you see &#x201C;Logstash startup completed&#x201D; type in your favorite test term on the terminal. Mine is &#x201C;fisk&#x201D; so I type that. You should see output on your screen showing what Logstash intends to pass on to Elasticsearch.
</p>
<p>We want to make sure this actually hits the cloud, so open a browser window and paste the HTTPS link from before, append <code>/_search</code> to the URL and hit enter. You should now see the search results from your newly created Elasticsearch cluster, containing the favorite term you just typed in. We have a functioning connection from Logstash on our machine to Elasticsearch in the cloud! Congratulations!
</p>
</section><section id="configuring-kibana-4" class="level2">
<h2><a href="#configuring-kibana-4">Configuring Kibana 4</a></h2>
<p>Kibana 4 comes with a built-in webserver. The configuration is done in a kibana.yml file in the config directory. Connecting to Elasticsearch in the cloud comes down to inserting the address of the Elasticsearch instance.
</p>
<pre class="prettyprint"># The Elasticsearch instance to use for all your queries.
elasticsearch_url: `https://....foundcluster.com:9243`
</pre>
<p>Of course, we need to verify that this really works, so we open up Kibana on <a href="http://localhost:5601">http://localhost:5601</a>, select the Logstash index template, with the <code>@timestamp</code> data field as suggested, and open up the discover panel. Now, if there was less than 15 minutes since you inserted your favorite test term in Logstash (previous step), you should see it already. Otherwise, change the date range by clicking on the selector in the top right corner.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c979985d58976376bf3/download?uid=blt0a8a562890429fb6?uid=blt0a8a562890429fb6" data-sys-asset-uid="blt806339984d59ea2d" alt="Kibana test" style="max-width:100%;">
</p><figcaption>Kibana test</figcaption>
</section><section id="locking-it-down" class="level2">
<h2><a href="#locking-it-down">Locking it down</a></h2>
<p>Found by Elastic has worked hard to make the previous steps easy. We created an Elasticsearch cluster, fed data into it and displayed in Kibana in less than 5 minutes. We must have forgotten something!? And yes, of course! Something about security. We made sure to use secure connections with SSL, and the address generated for our cluster contains a 32 character long, randomly generated list of characters, which is pretty hard to guess. Should, however, the address slip out of our hands, hackers could easily delete our entire cluster. And we don&#x2019;t want that to happen. So let&#x2019;s see how we can make everything work when we add some basic security measures.
</p>
</section><section id="access-control-lists" class="level2">
<h2><a href="#access-control-lists">Access Control Lists</a></h2>
<p>Found by Elastic has support for access control lists, where you can set up lists of usernames and passwords, with lists of rules that deny/allow access to various paths within Elasticsearch. This makes it easy to create a &#x201C;read only&#x201D; user, for instance, by creating a user with a rule that only allows access to the <code>/_search</code> path. Found by Elastic has a sample configuration with users searchonly and readwrite. We will use these as starting point but first we need to figure out what Kibana needs.
</p>
</section><section id="kibana-4-security" class="level2">
<h2><a href="#kibana-4-security">Kibana 4 Security</a></h2>
<p>Kibana 4 stores its configuration in a special index, by default named &#x201C;.kibana&#x201D;. The Kibana webserver needs write access to this index. In addition, all Kibana users need write access to this index, for storing dashboards, visualizations and searches, and read access to all the indices that it will query. More details about configuring Kibana 4 can be found in the article <a href="/guide/en/kibana/current/production.html">Using Kibana in a Production Environment</a>, for details on the access demands of Kibana, see <a href="/guide/en/shield/current/_shield_with_kibana_4.html#kibana4-roles">Configuring Roles for Kibana 4 Users</a>.
</p>
<p>For this demo, we will simply copy the &#x201C;readwrite&#x201D; user from the sample twice, naming one kibanaserver, the other kibanauser.
</p>
<pre class="prettyprint"># Setting Access Control in Found: 
# Allow everything for the readwrite-user, kibanauser and kibanaserver
  - paths: [&apos;.*&apos;]
    conditions:
      - basic_auth:
          users:
            - readwrite
            - kibanauser
            - kibanaserver
      - ssl:
          require: true
    action: allow
</pre>
<p>Press save and the changes are immediately effective. Try to reload the Kibana at <a href="http://localhost:5601">http://localhost:5601</a>, you should be denied access.
</p>
<p>Open up the kibana.yml file from before and modify it:
</p>
<pre class="prettyprint"># If your Elasticsearch is protected with basic auth, this is the user credentials
# used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana
# users will still need to authenticate with Elasticsearch (which is proxied thorugh
# the Kibana server)
kibana_elasticsearch_username: kibanaserver
kibana_elasticsearch_password: `KIBANASERVER_USER_PASSWORD`
</pre>
<p>Stop and start Kibana to effectuate settings. Now when Kibana starts up, you will be presented with a login box for HTTP authentication. Type in kibanauser as the username, and the password. You should now again be presented with the Discover screen, showing the previously entered favorite test term. Again, you may have to expand the time range to see your entry.
</p>
</section><section id="logstash-security" class="level2">
<h2><a href="#logstash-security">Logstash Security</a></h2>
<p>Logstash will also need to supply credentials when connecting to Found by Elastic. We reuse permission from the readwrite user once again, this time giving the name &#x201C;logstash&#x201D;. It is simply a matter of supplying the username and password in the configuration file.
</p>
<pre class="prettyprint">output {
    elasticsearch {
     &#x2026;. 
    user =&gt; &#x201C;logstash&#x201D;,
    password =&gt; `LOGSTASH_USER_PASSWORD`
    }
}
</pre>
</section><section id="wrapping-it-up" class="level2">
<h2><a href="#wrapping-it-up">Wrapping it up</a></h2>
<p>This has been a short dive into Logstash and Kibana with Found by Elastic. The recent changes done in order to support the Shield plugin for Elasticsearch, Logstash and Kibana, make it very easy to use the secure features of Found by Elastic. In the <a href="/blog/found-analyzing-weblogs-with-elasticsearch-part-2">next post</a> we will look into feeding logs from IIS into Elasticsearch via Logstash, and visualizing the most used query terms in Kibana.
</p>
</section></body></html>

Analyzing Weblogs with Elasticsearch in the Cloud, Part 1

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><h2>Using Logstash and Kibana on Found by Elastic</h2><p>This is part two in the series about using Elasticsearch, Kibana and Logstash with Found by Elastic. In the <a href="/blog/found-analyzing-weblogs-with-elasticsearch">previous post</a>, we used Logstash 1.5 and Kibana 4.02 to communicate with Elasticsearch in the cloud. This time we will use Logstash to feed logs from a web search application running on the Microsoft web server IIS (Internet Information Service) into Elasticsearch, and use Kibana to show some nice graphs.
</p><section id="logstash" class="level2"><a href="/blog/found-analyzing-weblogs-with-elasticsearch">
</a>
<h2><a href="/blog/found-analyzing-weblogs-with-elasticsearch"></a><a href="#logstash">Logstash</a></h2>
<p>Logstash is a tool for pipeline processing of logs. It can read logs from many different sources, has a flexible processing framework, and a wide selection of outputs. In this post we will read input from a file, split the raw text line into fields, before finally the processed and clean fields are sent off into Elasticsearch.
</p>
<p>Logstash comes with an extensive set of <a href="https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns">pre built log patterns</a>. IIS, unlike Apache, doesn&#x2019;t have a standard log format, making this a nice opportunity to create our own <a href="/guide/en/logstash/current/plugins-filters-grok.html#_grok_basics">grok pattern</a> to parse it.
</p>
<pre class="prettyprint"> grok {
    # TODO: check that this pattern matches your IIS log settings
    match =&gt; [&quot;message&quot;, &quot;%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{NOTSPACE:method} %{URIPATH:page} (?:%{NOTSPACE:querystring}|-) %{NUMBER:port} (%{WORD:username}|-) %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}&quot;]
  }
  #Set the Event Timestamp from the log
  date {
    match =&gt; [ &quot;log_timestamp&quot;, &quot;YYYY-MM-dd HH:mm:ss&quot; ]
    timezone =&gt; &quot;Etc/UCT&quot;
  }
</pre>
<p>The IIS logs are from a search application and contain the search terms in a query string parameter called, of course, <code>query</code>. The <code>kv</code> filter is used to split the query string into key value pairs, and we also need to replace URL encoding using the <code>urldecode</code> filter.
</p>
<pre class="prettyprint">  #Split querystring into key-values
  kv {
    source =&gt; &quot;querystring&quot;
    field_split =&gt; &quot;&amp;&quot;
  }
 #fix url encoding of query
  urldecode  {
    field =&gt; &quot;query&quot;
  }
</pre>
<p>The full config file for IIS can be found in <a href="https://gist.github.com/babadofar/5fbea416c6a07ca209bf">this gist</a>
</p>
<p>Again, there is no standard defined format for IIS logs, so while this grokked my logs, it might not grok yours.
</p>
</section><section id="grok-development" class="level2">
<h2><a href="#grok-development">Grok Development</a></h2>
<p>One of the most convenient ways to create grok patterns is <a href="http://grokconstructor.appspot.com/do/construction">the Grok Constructor</a>. Paste in some sample log lines, and press the &#x201C;Go!&#x201D; button. A long list of suggested matches for the first field is displayed. Select the best match for the first field matching your data. Keep on selecting until the whole line is covered. If you need more tips on how to work with developing Logstash configurations, see <a href="http://blog.comperiosearch.com/blog/2015/04/10/how-to-develop-logstash-configuration-files/">this blog post</a>.
</p>
</section><section id="running-logstash" class="level2">
<h2><a href="#running-logstash">Running Logstash</a></h2>
<p>Download the gist file and save it as <code>iisClient.conf</code>. Modify the file location, Elasticsearch host, user and password to match your setup. Wildcards for the file location should work fine using Logstash 1.5. Run it with this command:
</p>
<pre class="prettyprint">logstash.bat agent --verbose -f iisClient.conf
</pre>
<p>Logstash is normally running relatively quietly &#x2013; I like to use the <code>--verbose</code> flag to get a bit more detail on what&#x2019;s happening. You can use <code>--debug</code> if you&#x2019;re really curious. This configuration outputs both Elasticsearch and writes debug messages to the console with the rubydebug codec. When developing Logstash configurations, it&#x2019;s quite handy just to output to the console. Once the configuration is settled, disable the stdout output. Unless you&#x2019;d really like to watch screens with fast moving text.
</p>
</section><section id="kibana" class="level2">
<h2><a href="#kibana">Kibana</a></h2>
<p>You can start up Kibana while Logstash is running, to see the progress as Logstash works. On opening Kibana, a welcome screen presents options for selecting an index pattern and a timestamp. For this project, select the default suggested index pattern and the default <code>@timestamp</code> field for the Time-field name. If you don&#x2019;t see any data, either nothing reached Elasticsearch or you need to expand the date range from the datepicker in the upper right corner.
</p>
<p>Clicking on the <a href="/guide/en/kibana/current/discover.html">Discover</a> tab shows a list of search results, the number of hits on the right side, and on the left a list of fields from your documents. Clicking on a field reveals some quick statistics on the contents, gathered from a random sample of 500 documents.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c970342dfd7382632b2/download?uid=blta53540e64344323c?uid=blta53540e64344323c" data-sys-asset-uid="blt0749137e85edd406" alt="FieldStats" style="max-width:100%;">
</p><figcaption>FieldStats</figcaption>
</section><section id="graphing-most-used-query-terms" class="level2">
<h2><a href="#graphing-most-used-query-terms">Graphing Most Used Query Terms</a></h2>
<p>For a search application, the most interesting metric is perhaps the most used query terms. Select &#x201C;query&#x201D; from the list of fields. Clicking on the &#x201C;Visualize&#x201D; button at the bottom of the field metrics opens up the Visualize tab with a bar chart of the most used query terms. If you want the analysis to show complete queries (not broken up into terms), select the &#x201C;query.raw&#x201D; field rather than &#x201C;query&#x201D;. Logstash by default indexes all fields twice, using the default analyzer, and <code>not_analyzed</code>. For analyzed fields Elasticsearch splits the field contents into individual terms (tokens), which is very useful when searching. However, when analyzing user behavior and dealing with statistics, you often need the complete query string given by the user. This will be available in the not analyzed version of the field, which you can access by using the &#x201C;field.raw&#x201D; notation.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c979985d58976376bf8/download?uid=blt107ab41701c3df6e?uid=blt107ab41701c3df6e" data-sys-asset-uid="blt808d1a30a54dcafd" alt="Term Raw Selector" style="max-width:100%;">
</p><figcaption>Term Raw Selector</figcaption>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c97d8edd48f76935f68/download?uid=blt54fda33227632fad?uid=blt54fda33227632fad" data-sys-asset-uid="blt3969a779fc5ec602" alt="Most used queries" style="max-width:100%;">
</p><figcaption>Most used queries</figcaption>
<p>The graph shows an interesting phenomenon often seen where natural language is used. The distribution of search queries normally follow something similar to <a href="http://en.wikipedia.org/wiki/Zipf%27s_law">Zipf&#x2019;s law</a>, or the <a href="http://en.wikipedia.org/wiki/Pareto_principle">80/20 rule</a>, where 20% of the queries account for 80% of traffic.
</p>
</section><section id="graphing-number-of-requests" class="level2">
<h2><a href="#graphing-number-of-requests">Graphing Number of Requests</a></h2>
<p>To create a graph showing the number of requests per time-slot, enter the Visualize panel and press the little button to create a new visualization. Select &#x201C;Line chart&#x201D; as type of visualization, and &#x201C;from a new search&#x201D;. Press the &#x201C;Add aggregation&#x201D; button, and select Aggregation type &#x201C;Date histogram&#x201D;, and the field <code>@timestamp</code>. Press Apply at the bottom. If you have data (in the applied date range), you should see a graph similar to this one.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c973dc542cb38c08442/download?uid=blt5655981d28e5b228?uid=blt5655981d28e5b228" data-sys-asset-uid="bltc9e13643116a8835" alt="Requests pr second" style="max-width:100%;">
</p><figcaption>Requests pr second</figcaption>
</section><section id="graphing-response-times" class="level2">
<h2><a href="#graphing-response-times">Graphing Response Times</a></h2>
<p>If you looked closely at the Logstash configuration, you will find that it contains a section where a field &#x201C;time_taken&#x201D; is converted into integer. By default, Logstash will output all fields as of type string. To work properly with numeric ranges, like the Histogram aggregation, we have to be explicit about making sure the field is of a numeric type. Again, start a new visualization, of type &#x201C;Line Chart&#x201D;, &#x201C;from a new search&#x201D;. Now we are going to modify the Y-Axis, setting the type of Aggregation to Average, and using the field &#x201C;time_taken&#x201D; as input. For X-Axis we will use the Date histogram and the <code>@timestamp</code> field. Now you should see a graph displaying how the average response times varies over time.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9758208ba076e28202/download?uid=blte777248f776707bc?uid=blte777248f776707bc" data-sys-asset-uid="blt9e83201d32ad6ee5" alt="Average Response Time" style="max-width:100%;">
</p><figcaption>Average Response Time</figcaption>
</section><section id="summary" class="level2">
<h2><a href="#summary">Summary</a></h2>
<p>This has been a short guide to how you can get an overview of user engagement and application performance. The ELK stack offers a flexible toolset for analyzing and visualizing data - these steps are just scratching the surface. Hopefully, this will be enough to start your own journey into the realm of the ELK stack.
</p>
<p>If you feel like learning more, here are some useful links:
</p>
<p><a href="/blog/kibana-4-video-tutorials-part-1">Kibana video tutorials part 1</a>
</p>
<p><a href="/blog/kibana-4-video-tutorials-part-2">Kibana video tutorials part 2</a>
</p>
<p><a href="/guide/en/kibana/current/index.html">Kibana User Guide</a>
</p>
<p><a href="/guide/en/logstash/current/index.html">Logstash Reference</a>
</p>
<p>And last - to be able to use Kibana properly, it really helps to know how aggregations work:
</p>
<p><a href="/guide/en/elasticsearch/guide/current/aggregations.html">Elasticsearch Guide - Aggregations</a>
</p>
</section></body></html>

Analyzing Weblogs with Elasticsearch in the Cloud, Part 2

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Christoffer Vig

<p>Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.</p>

Newsletter GDPR Text - Ukrainian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

Intentionally Blank

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。</p>

Newsletter GDPR Text - TW

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p><p>By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.</p>

GDPR Text + Google Cloud

<p>Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.</p>

Newsletter GDPR Text - Italian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Christoffer Vig

Analyzing Weblogs with Elasticsearch in the Cloud, Part 1

Analyzing Weblogs with Elasticsearch in the Cloud, Part 2