Ever wonder how your users are using your Elasticsearch cluster? Have you felt the need to investigate the queries sent to the Elasticsearch cluster by your users?
Using Packetbeat you can keep an eye on what comes and goes and avoid those nasty surprises your users may throw at your cluster.
You can use Elastic plugins to monitor your own Elastic Stack, with the addition of <a href="/what-is/elasticsearch-monitoring">Marvel</a> the road to great monitoring started and is continuing to improve in the future generation as part of the <a href="/downloads/x-pack">X-Pack</a> bundle. Long term, this will best provide insight into cluster performance, even behind SSL. Elastic recommends that you make use of these components first and foremost, and that you do have a dedicated Elasticsearch cluster running on at least one node, for this purpose.&nbsp;This monitoring cluster is a great place to also store additional query detail; so if you don't already have a monitoring cluster this gives you another great reason to set it up as it is imperative that you send the query data to a separate cluster.
Please note this blog focuses on monitoring search traffic over HTTP only; current versions of Packetbeat do not support inspecting encrypted payloads
<h3>What’s a monitoring cluster?</h3>A monitoring cluster is a cluster dedicated for storing and analyzing the monitoring data from your production Elasticsearch cluster. Keeping your monitoring data on a separate cluster is highly recommended; if things do go wrong in production, you want insight to this data and you want them somewhere you can access them (outside the “fire zone”).
This separation becomes essential if you are planning to monitor search queries via Packetbeat.
If you have Marvel but this is you wish to set up a monitoring cluster this <a href="/guide/en/marvel/current/installing-marvel.html">document</a> is a good starting point. Make sure to give your monitoring cluster a face by installing a dedicated Kibana instance for monitoring.
<h3>Picture it</h3><h3><img src="https://api.contentstack.io/v2/assets/57ac7809bc5ccd727e689158/download?uid=blt29de971b7c15e4d9" data-sys-asset-uid="blt29de971b7c15e4d9" alt="Monitoring Search Queries-final.jpg"></h3><h3>Getting ready for the data </h3><ol>
	<li> Install a Logstash instance to process your monitored packets.</li>
</ol>You’ll need this for to filter the traffic to specific portions of the traffic you are interested (i.e., only search queries).
<a href="/guide/en/logstash/current/installing-logstash.html">https://www.elastic.co/guide/en/logstash/current/installing-logstash.html</a>
<ol start="2">
	<li>Configure Logstash. For this example I chose to only monitor search queries.</li>
</ol>I created config file called sniff_search.conf with below content; it includes extracting query_body and the index that has been searched into their own fields. You can go as crazy as you wish here with extracting bits that are useful to you.
<pre class="prettyprint">input {
 beats {
 port =&gt; 5044
 }
}
filter {
 if "search" in [request]{
 grok {
 match =&gt; { "request" =&gt; ".*\n\{(?&lt;query_body&gt;.*)"} 
 }
 grok {
 match =&gt; { "path" =&gt; "\/(?&lt;index&gt;.*)\/_search"} 
 }
 if [index] {
 } 
 else {
 mutate {
 add_field =&gt; { "index" =&gt; "All" }
 }
 }
 mutate {
 update =&gt; { "query_body" =&gt; "{%{query_body}" }
 }
 }
}
output {
 if "search" in [request] and "ignore_unmapped" not in [query_body]{
 elasticsearch {
 hosts =&gt; "10.255.4.165:9200"
 }
 }
}
</pre><ol start="3">
	<li>Start Logstash:</li>
</ol> On Linux:
<pre class="prettyprint">./bin/logstash -f sniff_search.conf
</pre> <a href="/guide/en/beats/libbeat/current/logstash-installation.html#_starting_logstash">https://www.elastic.co/guide/en/beats/libbeat/current/logstash-installation.html#_starting_logstash</a>
<h3>Start sniffing</h3><ol>
	<li> Install Packetbeat on each host of the production cluster you would like to monitor.</li>
	<li>Configure packetbeat.yml on each node</li>
</ol><ul>
	<li>Change the interface to any or configure a specific network interface, when using specific interface this has to match what Elasticsearch is binding to.</li>
	<li>Please note you can use “device: any” only on Linux kernel. For OSX to listen to 9200 port, please specify "device:lo0", and for Windows, please&nbsp;<a href="/guide/en/beats/packetbeat/current/faq.html#packetbeat-loopback-interface">refer to the documentation</a>&nbsp;for the correct device to include.</li>
</ul><pre class="prettyprint"># Select the network interfaces to sniff the data. You can use the "any"
# keyword to sniff on all connected interfaces.
 interfaces:
 device: any
</pre><ul>
	<li>Configure the ports where to listen for HTTP traffic, by default Elasticsearch uses port 9200 for HTTP traffic. </li>
</ul><pre class="prettyprint">http: 
# Configure the ports where to listen for HTTP traffic. You can disable 
# the HTTP protocol by commenting out the list of ports. 
 ports: [9200] 
 send_request: true 
 include_body_for: ["application/json", "x-www-form-urlencoded"]
</pre><ul>
	<li>Comment out the port keys for other protocols to disable them.</li>
	<li>Configure Packetbeat to send the data to the monitoring Logstash instance. Note to comment out the default Elasticsearch output.</li>
</ul><pre class="prettyprint">#elasticsearch:
 # Array of hosts to connect to.
 # Scheme and port can be left out and will be set to the default (http and 9200)
 # In case you specify and additional path, the scheme is required: http://localhost:9200/path
 # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
 # hosts: ["Localhost:9200"]
 ### Logstash as output
logstash:
 # The Logstash hosts
 hosts: ["10.255.4.166:5044"]
</pre><ol start="3">
	<li>Start Packetbeat. For sniffing the packets it must be started as root.</li>
</ol>On Linux:
<pre class="prettyprint">sudo ./packetbeat -e -c packetbeat.yml -d "publish"
</pre> <a href="/guide/en/beats/packetbeat/current/_step_4_starting_packetbeat.html">https://www.elastic.co/guide/en/beats/packetbeat/current/_step_4_starting_packetbeat.html</a>
<ol start="4">
	<li>After starting Packetbeat it will listen for packets on 9200 sending them to Logstash and from there to the monitoring Elasticsearch cluster, it will be indexed in indexes like: logstash-2016.05.24</li>
</ol>This is an example of what a document will look like:
<pre class="prettyprint">{
 "bytes_in" =&gt; 537,
 "client_ip" =&gt; "10.255.5.101",
 "client_port" =&gt; 52213,
 "client_proc" =&gt; "",
"client_server" =&gt; "",
 "ip" =&gt; "10.255.4.167",
 "port" =&gt; 9200,
 "path" =&gt; "/logstash-*/_search",
 "beat" =&gt; {
 "hostname" =&gt; "ip-10-255-4-167.eu-west-1.compute.internal",
 "name" =&gt; "ip-10-255-4-167.eu-west-1.compute.internal"
},
 "proc" =&gt; "",
 "server" =&gt; "",
 "method" =&gt; "POST",
 "type" =&gt; "http",
 "status" =&gt; "OK",
 "params" =&gt; "%7B+%22query%22%3A+%7B%0A%22match%22%3A+%7B%0A+++%22clientip%22%3A+%22105.235.130.196%22%0A%7D%0A%7D%7D%0A=",
 "http" =&gt; {
 "code" =&gt; 200,
 "content_length" =&gt; 7587,
 "phrase" =&gt; "OK"
},
 "bytes_out" =&gt; 7675,
 "request" =&gt; "POST /logstash-*/_search HTTP/1.1\r\nHost: 10.255.4.167:9200\r\nConnection: keep-alive\r\nContent-Length: 62\r\nAccept: application/json, text/javascript, */*; q=0.01\r\nOrigin: chrome-extension://lhjgkmllcaadmopgmanpapmpjgmfcfig\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36\r\nContent-Type: application/x-www-form-urlencoded; charset=UTF-8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-US,en;q=0.8\r\n\r\n{ \"query\": {\n\"match\": {\n \"clientip\": \"105.235.130.196\"\n}\n}}\n",
 "@timestamp" =&gt; "2016-08-05T14:35:36.740Z",
 "query" =&gt; "POST /logstash-*/_search",
 "count" =&gt; 1,
 "direction" =&gt; "in",
 "responsetime" =&gt; 28,
 "@version" =&gt; "1",
 "host" =&gt; "ip-10-255-4-167.eu-west-1.compute.internal",
 "tags" =&gt; [
 [0] "beats_input_raw_event"
 ],
 "query_body" =&gt; "{ \"query\": {\n\"match\": {\n \"clientip\": \"105.235.130.196\"\n}\n}}\n",
 "index" =&gt; "logstash-*"
}
</pre>From the data you see the IP and the port of the client connected to Elasticsearch "client_ip": "10.255.5.101" , "client_port": 56433. The IP of the node and port &nbsp;&nbsp;"ip": "10.255.4.167", "port": 9200,, you can also see the query sent to Elasticsearch "query": "POST /logstash-*/_search",...
You can visualize this data in Kibana. Connect to Kibana you installed in step 2 and configure an index pattern of Logstash-*
You can find an example of visualization you can create below:
<img src="https://api.contentstack.io/v2/assets/57ac7c03bc5ccd727e689321/download?uid=blta05c984567f25af5" data-sys-asset-uid="blta05c984567f25af5" alt="dashboard copy.jpg">
Above dashboard is just one example you might find useful, if interested you can download it <a href="https://gist.github.com/JalehD/d85f85a1aa3bbe534bf0aec0a9c69777">here</a>.
You can easily import the visualizations and the dashboard via Kibana through Settings &gt; Objects tab.
<img src="https://api.contentstack.io/v2/assets/57ac7c23bc5ccd727e689328/download?uid=blt4d79dc2785b42220" data-sys-asset-uid="blt4d79dc2785b42220" alt="Screen Shot 2016-08-08 at 14.59.58.png">
<h3>When/why monitor search queries</h3>Maintaining a healthy cluster needs insight into how it’s used and the search queries your users run. Whether you provide direct search access to the cluster to your users or have an application layer in between, usage patterns can be helpful in planning for your data and resources.
Once you have the data inside your monitoring cluster you can answer questions like these and many more:
<ol>
	<li>What are the most used queries?</li>
	<li>What are my most searched indexes?</li>
	<li>Does a high average search response time period correlate with other monitoring information from Marvel? </li>
	<li>What are the slowest queries?</li>
	<li>Are the slow queries on a particular index; maybe a deeper look into that index’s settings is needed?</li>
	<li>Number of searches from clients, are there abnormal peaks in usage from a specific client?</li>
</ol>Answering these questions helps better planning and outage prevention.
[1]<a href="/downloads/elasticsearch"> https://www.elastic.co/downloads/elasticsearch</a>
[2]<a href="/downloads/kibana"> https://www.elastic.co/downloads/kibana</a>

test2.jpg

blog-monitor-queries-thumbnail.jpg

Monitoring the Search Queries

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Jaleh Dastmalchi-Round

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、上記で提供された連絡先に<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">特定の通知</a>が送信されることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシーポリシー</a>をご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。このフォームを送信することにより、Elasticが私の連絡先情報をGoogle Cloudに共有することに同意します。また、Elasticが私に食材パッケージを発送する目的で私の情報をベンダーに提供することにも同意します。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

送信することにより、Elasticの<a href="/jp/legal/serverless-preview-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

送信することにより、<a href="/jp/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud利用規約</a>に同意したものとみなされます。個人情報は<a href="/jp/legal/privacy-statement">Elasticのプライバシーポリシー</a>に基づいて取り扱われます。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

送信することにより、Elasticの<a href="/jp/legal/terms-of-use" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

このページのコンテンツは、選択された言語ではご利用いただけません。Elasticでは現在、コンテンツをさまざまな言語でご利用いただけるよう取り組んでいます。お使いの言語が準備されるまでお待ちくださるようお願いいたします。

Author

Articles by

Watch now (no PT)

Watch now

See all top stories

すべてのトップストーリーを見る

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Search Integrations

All Solutions

Video for

See when this webinar starts in my time zone

現在のタイムゾーンでこのウェビナー開始時間を見る

Register to Watch

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

他の予定と重なってしまった…という場合も、問題ありません。参加登録されると、録画配信をご利用いただけます。関連コンテンツのご案内もメールでお届けします。

Author

Articles by Jaleh Dastmalchi-Round

Monitoring the Search Queries

Sign up for Elastic Cloud free trial