<html><head></head><body>Like every other software out there, Elasticsearch provides some insight into what&apos;s happening with its internals while running. This is an informative process and can reveal quickly if something bad happened (in the form of an error message), or it can decisively determine the cause of a performance issue or why the cluster is behaving in a way users might not like.

<h2>The Basics</h2>
Main Elasticsearch logs are written to <code>ES_HOME/logs/[cluster_name].log</code> file. For this file the default level is INFO, thus being sufficient for a rather moderate amount of information and, at the same time, not create a huge log file.

Logging settings are specified in the logging.yml configuration file. While this is a convenient way of having the log settings in a single place, having to modify this file on every node every time you want to change the settings is painful because it requires nodes restart and unnecessary downtime. Good news! Elasticsearch allows you to &#xA0;<a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/logging.html#logging">update the logging settings dynamically</a>. The same applies to &#xA0;the second type of logging that it provides: slowlogs. These provide a very useful insight into how fast or slow queries/indexing process/fetches are performing. The slowlog file will show the query itself, on which shards it ran and how much time it took to complete.

For the logs, there is a list of &quot;levels&quot; which indicate how granular the logs will be. In the order of granularity from low to high, we have: WARN, INFO, DEBUG, TRACE. Setting the log level to WARN will display only WARN logging messages. On the other hand, setting the level to INFO will display both WARN and INFO logging messages, so this is a &quot;cumulative&quot; set of log messages. Similarly, for TRACE every log message in Elasticsearch will be printed in the log files. Setting the level to TRACE will create a considerable amount of chatter.

<h2>Dynamically changing the root logger level</h2>
This is not necessarily a tricky task :-), but for completeness sake and to warm up before the following tips&amp;tricks let&apos;s first change the logging level for the root logger (the one from which all other loggers inherit). You can run a query like the following, while the cluster is running of course:

<pre class="prettyprint">PUT /_cluster/settings
{&quot;transient&quot;:{&quot;logger._root&quot;:&quot;DEBUG&quot;}}
</pre>
Very simple and quick. Now you should see some more interesting things in the logs, some messages that weren&apos;t available with INFO or WARN levels.

<h2>Dynamically changing the logger level per Java package</h2>
Many times it is very useful to see logging messages that are related to a certain part or functionality of Elasticsearch. Increasing the logging level globally in all modules in Elasticsearch would result in a potentially very large log file, so Elasticsearch allows &#xA0;individual &quot;sections&quot; to&#xA0;be configured separately.

These &quot;sections&quot; are the Java packages and the granularity logging level can be set per Java package. Let&apos;s take, for example, the package <code>org.elasticsearch.discovery.zen</code>. It contains classes that deal with<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html"> Zen discovery in Elasticsearch</a>. If you have an issue with nodes joining/leaving the cluster, or with master election, this would be the package to monitor in your logs:

<pre class="prettyprint">PUT /_cluster/settings
{&quot;transient&quot;: {&quot;logger.discovery.zen&quot;:&quot;TRACE&quot;}}
</pre>
You notice in the command that &quot;logger&quot; is again present and, yes, it should always be. But, how did we choose &quot;discovery.zen&quot;. You remember I mentioned above that the logging level can be set per Java package. Well, &quot;discovery.zen&quot; is just that, without the already present &quot;org.elasticsearch&quot;. What if you are interested only in the master node fault detection messages? The Java package containing classes for master fault detection is&#xA0;<code>org.elasticsearch.discovery.zen.fd</code>. And the Elasticsearch command&#xA0;would be:

<pre class="prettyprint">PUT /_cluster/settings
{&quot;transient&quot;: {&quot;logger.discovery.zen.fd&quot;:&quot;TRACE&quot;}}
</pre>
You got the picture. Some ideas on what to monitor and when: 

<ul>
	<li dir="ltr"><code>org.elasticsearch.env</code> for logging messages referring to node&apos;s data paths</li>
	<li dir="ltr"><code>org.elasticsearch.indices.recovery</code> and &lt;code&gt;org.elasticsearch.index.gateway for logging messages referring to shards recoveries</li>
	<li dir="ltr"><code>org.elasticsearch.cluster.action.shard</code> for logging messages related to shard states (failed shards, shards starting etc)</li>
	<li dir="ltr"><code>org.elasticsearch.snapshots</code> for everything related to snapshots&amp;restore</li>
	<li dir="ltr"><code>org.elasticsearch.http</code> to monitor for any binding or connectivity/discovery issues between nodes</li>
	<li dir="ltr"><code>org.elasticsearch.marvel.agent.exporter</code> to monitor the Marvel agent</li>
</ul>
<h2>Slowlog settings at index level</h2>
You remember that I mentioned the logging settings can be dynamically changed, without nodes or cluster restart. This is valid for slow logs, as well. Imagine you have several tens of indices in your cluster and you suspect that one of the older indices is performing poorly. You&apos;d want to investigate this index more closely and monitor the query response times. The first approach would be to<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html#search-slow-log"> set the slowlog logging level to DEBUG or TRACE cluster-wide</a> and grep the resulting slowlog file for your index name.

This will work, indeed, but why having to grep for the desired index name? Instead, set the slowlog settings at index level, only for the index you are interested into:

<pre class="prettyprint">PUT /test_index/_settings
{
 &quot;index&quot;: {
 &quot;search.slowlog.level&quot;: &quot;trace&quot;,
 &quot;search.slowlog.threshold.query.trace&quot;: &quot;100ms&quot;
 }
}
</pre>
The settings above change the slowlog logging level and the threshold for only the &#xA0;test_index. In this way you can monitor only the queries for that particular index without the overhead of enabling &quot;trace&quot; logging for all indices.

<h2>Logging inside Groovy scripts</h2>
Another trick that I found myself using several times to debug Groovy scripts in Elasticsearch is logging inside the script itself to double check that the script is doing what I believe it&apos;s doing. For example, you have an index with documents that hold a timestamp and you want to filter your documents considering a certain week of the year. Using a script this goes like this:

<pre class="prettyprint">{
 &quot;query&quot;: {
 &quot;filtered&quot;: {
 &quot;filter&quot;: {
 &quot;script&quot;: {
 &quot;script&quot;: &quot;weeks.contains((int)doc[&apos;timestamp&apos;].date.toDateTime().getWeekOfWeekyear())&quot;,
 &quot;params&quot;: {
 &quot;weeks&quot;: [
 18,19,20
 ]
 },
 &quot;lang&quot;: &quot;groovy&quot;
 }
 }
 }
 }
}
</pre>
To check if the correct Groovy method for finding out the week of the year is <code>getWeekOfWeekyear()</code>, this logging can be added to the script, to print in the logs the actual value returned by that method:

<pre class="prettyprint">import org.elasticsearch.common.logging.*;
ESLogger logger = ESLoggerFactory.getLogger(&apos;myscript&apos;);
logger.logger.log(org.apache.log4j.Level.INFO, (int)doc[&apos;timestamp&apos;].date.toDateTime().getWeekOfWeekyear());
</pre>
And the final query becomes:

<pre class="prettyprint">{
 &quot;query&quot;: {
 &quot;filtered&quot;: {
 &quot;filter&quot;: {
 &quot;script&quot;: {
 &quot;script&quot;: &quot;import org.elasticsearch.common.logging.*; ESLogger logger=ESLoggerFactory.getLogger(&apos;myscript&apos;); logger.logger.log(org.apache.log4j.Level.INFO,(int)doc[&apos;timestamp&apos;].date.toDateTime().getWeekOfWeekyear());values.contains((int)doc[&apos;timestamp&apos;].date.toDateTime().getWeekOfWeekyear())&quot;,
 &quot;params&quot;: {
 &quot;values&quot;: [
 18,19,20
 ]
 },
 &quot;lang&quot;: &quot;groovy&quot;
 }
 }
 }
 }
}
</pre>
 

<h2>Deprecation Logging in 2.0 </h2>
Among many other planned goodies in the 2.0 release of Elasticsearch, we also added a<a href="https://github.com/elastic/elasticsearch/pull/11033"> new deprecation log</a> to notify users when they use any settings, features, parameters etc which have been deprecated. 
 </body></html>

Elasticsearch Logging Secrets

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Andrei Stefan

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Andrei Stefan

Elasticsearch Logging Secrets

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS