Ever wonder how your users are using your Elasticsearch cluster? Have you felt the need to investigate the queries sent to the Elasticsearch cluster by your users?
Using Packetbeat you can keep an eye on what comes and goes and avoid those nasty surprises your users may throw at your cluster.
You can use Elastic plugins to monitor your own Elastic Stack, with the addition of <a href="/what-is/elasticsearch-monitoring">Marvel</a> the road to great monitoring started and is continuing to improve in the future generation as part of the <a href="/downloads/x-pack">X-Pack</a> bundle. Long term, this will best provide insight into cluster performance, even behind SSL. Elastic recommends that you make use of these components first and foremost, and that you do have a dedicated Elasticsearch cluster running on at least one node, for this purpose.&nbsp;This monitoring cluster is a great place to also store additional query detail; so if you don't already have a monitoring cluster this gives you another great reason to set it up as it is imperative that you send the query data to a separate cluster.
Please note this blog focuses on monitoring search traffic over HTTP only; current versions of Packetbeat do not support inspecting encrypted payloads
<h3>What’s a monitoring cluster?</h3>A monitoring cluster is a cluster dedicated for storing and analyzing the monitoring data from your production Elasticsearch cluster. Keeping your monitoring data on a separate cluster is highly recommended; if things do go wrong in production, you want insight to this data and you want them somewhere you can access them (outside the “fire zone”).
This separation becomes essential if you are planning to monitor search queries via Packetbeat.
If you have Marvel but this is you wish to set up a monitoring cluster this <a href="/guide/en/marvel/current/installing-marvel.html">document</a> is a good starting point. Make sure to give your monitoring cluster a face by installing a dedicated Kibana instance for monitoring.
<h3>Picture it</h3><h3><img src="https://api.contentstack.io/v2/assets/57ac7809bc5ccd727e689158/download?uid=blt29de971b7c15e4d9" data-sys-asset-uid="blt29de971b7c15e4d9" alt="Monitoring Search Queries-final.jpg"></h3><h3>Getting ready for the data </h3><ol>
	<li> Install a Logstash instance to process your monitored packets.</li>
</ol>You’ll need this for to filter the traffic to specific portions of the traffic you are interested (i.e., only search queries).
<a href="/guide/en/logstash/current/installing-logstash.html">https://www.elastic.co/guide/en/logstash/current/installing-logstash.html</a>
<ol start="2">
	<li>Configure Logstash. For this example I chose to only monitor search queries.</li>
</ol>I created config file called sniff_search.conf with below content; it includes extracting query_body and the index that has been searched into their own fields. You can go as crazy as you wish here with extracting bits that are useful to you.
<pre class="prettyprint">input {
 beats {
 port =&gt; 5044
 }
}
filter {
 if "search" in [request]{
 grok {
 match =&gt; { "request" =&gt; ".*\n\{(?&lt;query_body&gt;.*)"} 
 }
 grok {
 match =&gt; { "path" =&gt; "\/(?&lt;index&gt;.*)\/_search"} 
 }
 if [index] {
 } 
 else {
 mutate {
 add_field =&gt; { "index" =&gt; "All" }
 }
 }
 mutate {
 update =&gt; { "query_body" =&gt; "{%{query_body}" }
 }
 }
}
output {
 if "search" in [request] and "ignore_unmapped" not in [query_body]{
 elasticsearch {
 hosts =&gt; "10.255.4.165:9200"
 }
 }
}
</pre><ol start="3">
	<li>Start Logstash:</li>
</ol> On Linux:
<pre class="prettyprint">./bin/logstash -f sniff_search.conf
</pre> <a href="/guide/en/beats/libbeat/current/logstash-installation.html#_starting_logstash">https://www.elastic.co/guide/en/beats/libbeat/current/logstash-installation.html#_starting_logstash</a>
<h3>Start sniffing</h3><ol>
	<li> Install Packetbeat on each host of the production cluster you would like to monitor.</li>
	<li>Configure packetbeat.yml on each node</li>
</ol><ul>
	<li>Change the interface to any or configure a specific network interface, when using specific interface this has to match what Elasticsearch is binding to.</li>
	<li>Please note you can use “device: any” only on Linux kernel. For OSX to listen to 9200 port, please specify "device:lo0", and for Windows, please&nbsp;<a href="/guide/en/beats/packetbeat/current/faq.html#packetbeat-loopback-interface">refer to the documentation</a>&nbsp;for the correct device to include.</li>
</ul><pre class="prettyprint"># Select the network interfaces to sniff the data. You can use the "any"
# keyword to sniff on all connected interfaces.
 interfaces:
 device: any
</pre><ul>
	<li>Configure the ports where to listen for HTTP traffic, by default Elasticsearch uses port 9200 for HTTP traffic. </li>
</ul><pre class="prettyprint">http: 
# Configure the ports where to listen for HTTP traffic. You can disable 
# the HTTP protocol by commenting out the list of ports. 
 ports: [9200] 
 send_request: true 
 include_body_for: ["application/json", "x-www-form-urlencoded"]
</pre><ul>
	<li>Comment out the port keys for other protocols to disable them.</li>
	<li>Configure Packetbeat to send the data to the monitoring Logstash instance. Note to comment out the default Elasticsearch output.</li>
</ul><pre class="prettyprint">#elasticsearch:
 # Array of hosts to connect to.
 # Scheme and port can be left out and will be set to the default (http and 9200)
 # In case you specify and additional path, the scheme is required: http://localhost:9200/path
 # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
 # hosts: ["Localhost:9200"]
 ### Logstash as output
logstash:
 # The Logstash hosts
 hosts: ["10.255.4.166:5044"]
</pre><ol start="3">
	<li>Start Packetbeat. For sniffing the packets it must be started as root.</li>
</ol>On Linux:
<pre class="prettyprint">sudo ./packetbeat -e -c packetbeat.yml -d "publish"
</pre> <a href="/guide/en/beats/packetbeat/current/_step_4_starting_packetbeat.html">https://www.elastic.co/guide/en/beats/packetbeat/current/_step_4_starting_packetbeat.html</a>
<ol start="4">
	<li>After starting Packetbeat it will listen for packets on 9200 sending them to Logstash and from there to the monitoring Elasticsearch cluster, it will be indexed in indexes like: logstash-2016.05.24</li>
</ol>This is an example of what a document will look like:
<pre class="prettyprint">{
 "bytes_in" =&gt; 537,
 "client_ip" =&gt; "10.255.5.101",
 "client_port" =&gt; 52213,
 "client_proc" =&gt; "",
"client_server" =&gt; "",
 "ip" =&gt; "10.255.4.167",
 "port" =&gt; 9200,
 "path" =&gt; "/logstash-*/_search",
 "beat" =&gt; {
 "hostname" =&gt; "ip-10-255-4-167.eu-west-1.compute.internal",
 "name" =&gt; "ip-10-255-4-167.eu-west-1.compute.internal"
},
 "proc" =&gt; "",
 "server" =&gt; "",
 "method" =&gt; "POST",
 "type" =&gt; "http",
 "status" =&gt; "OK",
 "params" =&gt; "%7B+%22query%22%3A+%7B%0A%22match%22%3A+%7B%0A+++%22clientip%22%3A+%22105.235.130.196%22%0A%7D%0A%7D%7D%0A=",
 "http" =&gt; {
 "code" =&gt; 200,
 "content_length" =&gt; 7587,
 "phrase" =&gt; "OK"
},
 "bytes_out" =&gt; 7675,
 "request" =&gt; "POST /logstash-*/_search HTTP/1.1\r\nHost: 10.255.4.167:9200\r\nConnection: keep-alive\r\nContent-Length: 62\r\nAccept: application/json, text/javascript, */*; q=0.01\r\nOrigin: chrome-extension://lhjgkmllcaadmopgmanpapmpjgmfcfig\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36\r\nContent-Type: application/x-www-form-urlencoded; charset=UTF-8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-US,en;q=0.8\r\n\r\n{ \"query\": {\n\"match\": {\n \"clientip\": \"105.235.130.196\"\n}\n}}\n",
 "@timestamp" =&gt; "2016-08-05T14:35:36.740Z",
 "query" =&gt; "POST /logstash-*/_search",
 "count" =&gt; 1,
 "direction" =&gt; "in",
 "responsetime" =&gt; 28,
 "@version" =&gt; "1",
 "host" =&gt; "ip-10-255-4-167.eu-west-1.compute.internal",
 "tags" =&gt; [
 [0] "beats_input_raw_event"
 ],
 "query_body" =&gt; "{ \"query\": {\n\"match\": {\n \"clientip\": \"105.235.130.196\"\n}\n}}\n",
 "index" =&gt; "logstash-*"
}
</pre>From the data you see the IP and the port of the client connected to Elasticsearch "client_ip": "10.255.5.101" , "client_port": 56433. The IP of the node and port &nbsp;&nbsp;"ip": "10.255.4.167", "port": 9200,, you can also see the query sent to Elasticsearch "query": "POST /logstash-*/_search",...
You can visualize this data in Kibana. Connect to Kibana you installed in step 2 and configure an index pattern of Logstash-*
You can find an example of visualization you can create below:
<img src="https://api.contentstack.io/v2/assets/57ac7c03bc5ccd727e689321/download?uid=blta05c984567f25af5" data-sys-asset-uid="blta05c984567f25af5" alt="dashboard copy.jpg">
Above dashboard is just one example you might find useful, if interested you can download it <a href="https://gist.github.com/JalehD/d85f85a1aa3bbe534bf0aec0a9c69777">here</a>.
You can easily import the visualizations and the dashboard via Kibana through Settings &gt; Objects tab.
<img src="https://api.contentstack.io/v2/assets/57ac7c23bc5ccd727e689328/download?uid=blt4d79dc2785b42220" data-sys-asset-uid="blt4d79dc2785b42220" alt="Screen Shot 2016-08-08 at 14.59.58.png">
<h3>When/why monitor search queries</h3>Maintaining a healthy cluster needs insight into how it’s used and the search queries your users run. Whether you provide direct search access to the cluster to your users or have an application layer in between, usage patterns can be helpful in planning for your data and resources.
Once you have the data inside your monitoring cluster you can answer questions like these and many more:
<ol>
	<li>What are the most used queries?</li>
	<li>What are my most searched indexes?</li>
	<li>Does a high average search response time period correlate with other monitoring information from Marvel? </li>
	<li>What are the slowest queries?</li>
	<li>Are the slow queries on a particular index; maybe a deeper look into that index’s settings is needed?</li>
	<li>Number of searches from clients, are there abnormal peaks in usage from a specific client?</li>
</ol>Answering these questions helps better planning and outage prevention.
[1]<a href="/downloads/elasticsearch"> https://www.elastic.co/downloads/elasticsearch</a>
[2]<a href="/downloads/kibana"> https://www.elastic.co/downloads/kibana</a>

test2.jpg

blog-monitor-queries-thumbnail.jpg

Monitoring the Search Queries

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Jaleh Dastmalchi-Round

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Jaleh Dastmalchi-Round

Monitoring the Search Queries

Sign up for Elastic Cloud free trial