We've spent a lot of time lately tracking down <a href="https://www.elastic.co/blog/memory-issues-well-remember">various memory issues</a> in the Elastic Cloud service. One in particular hit small clusters with relatively little activity and needed quite a bit of digging to figure out.&nbsp;The following applies to version 5.1.1 of the Elastic Stack, and has been fixed in newer versions.&nbsp;Each node gets a set amount of memory and we reserve about half of that for heap. The rest is used by the JVM and file caches. Normally the memory used by the JVM is constant, but we saw some instances where the total memory used by Elasticsearch kept slowly increasing without bounds until it hit the total limit. Sometimes it took days, but it was constantly climbing. When reaching the limit, the node runs out of memory and is rebooted.So that's a pretty clear sign of a memory leak. We analysed Elasticsearch with tools like Yourkit, MAT, etc., but we couldn't find anything in the heap that would explain this, but the RSS (resident set size) memory of the Elasticsearch process was increasing and increasing. We tried turning on Native Memory Tracking and saw that the memory increase was not in the heap at all.Native Memory Tracking (NMT) is a feature in the JVM which provides a way to look into the memory usage inside the JVM. It tracks all memory allocations, and groups it by usage. This provides both an overview of where memory goes and also details to where allocations happen.A report from NMT looks like this:<img src="https://api.contentstack.io/v2/uploads/591ae5ffce4b4c7d4d62b3f8/download?uid=bltfb9c3bdce9e6dd46" data-sys-asset-uid="bltfb9c3bdce9e6dd46" alt="native-memory-tracking.png" width="721" height="298" style="width: 721;height: 298;"/>However, even after analyzing the Elasticsearch process over a long time with NMT, the memory usage as measured by the OS was increasing a lot faster than NMT reported. This showed that the memory leak was outside of the JVM's tracking capabilities. We have a native memory leak! Enter tools to track down native memory allocations. Fortunately someone has described one way to <a href="http://www.evanjones.ca/java-native-leak-bug.html">do that with jemalloc</a>. Here are the steps I followed to run it with Elasticsearch:<pre dir="ltr">git clone https://github.com/jemalloc/jemalloc cd jemalloc ./autoconf.sh --enable-perf make dist make sudo make install </pre>Then I added these env variables before starting Elasticsearch: <pre dir="ltr">export LD_PRELOAD=/usr/local/lib/libjemalloc.so export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_final:true </pre>I was running version 5.1.1 of Elasticsearch with X-Pack, which exhibited the memory leak, and I also enabled X-Pack monitoring features just to get some light indexing going.&nbsp;Then I ran it overnight, because we've seen that this leak happens slowly.Afterwards, the following output was&nbsp;generated with:<pre dir="ltr">jeprof --show_bytes --gif /usr/lib/jvm/java-8-oracle/jre/bin/java jeprof*.heap &gt; /tmp/app-profiling.gif </pre>The output looked like this (scroll to the bottom):<img src="https://api.contentstack.io/v2/uploads/591ae56b6cdfbb1541d79404/download?uid=bltd8f13293a3fa9dca" data-sys-asset-uid="bltd8f13293a3fa9dca" alt="jemalloc.png" width="873" height="1126" style="width: 873;height: 1126;"/>On the bottom it says that only 80% of the memory is actually referenced by the JVM, so somehow we “lost” 20%. This is after running for 24 hours. When I started the process this was 97%, so the loss kept growing.In the details, you can see which method allocated memory. The biggest offenders seemed to be java.util.zip.Inflater and java.util.zip.Deflater which accounted for 18.2% of the "missing" memory. Inflater and Deflater are implemented using native functions, which allocates memory for buffers. If you forget to call end() on them, the memory is never freed. However, it implements a finalizer, which is supposed to run end() for you if you forget. But finalizers are only called on full garbage collection, and when all references are gone. Clusters with light usage almost never do Full GC, so in practice they&nbsp;never got cleaned up. We went through our usages of Inflater and Deflater&nbsp;using Yourkit and found some places that could potentially be forgetting to close them: <img src="https://api.contentstack.io/v2/uploads/591ae582d3e1a79b4dec62f1/download?uid=blt6d7b953091090264" data-sys-asset-uid="blt6d7b953091090264" alt="yourkit-deflatecompression.png" width="914" height="182" style="width: 914;height: 182;"/><img src="https://api.contentstack.io/v2/uploads/591ae592d3e1a79b4dec62f6/download?uid=blt71caceab495c5fdb" data-sys-asset-uid="blt71caceab495c5fdb" alt="yourkit-jarfile-1.png" width="912" height="426" style="background-color: initial;width: 912;height: 426;"/> Not all of those are actual leaks, they could still be in use, but we investigated them. This investigation lead to various fixes in both Lucene and Elasticsearch: <a href="https://issues.apache.org/jira/browse/LUCENE-7647"> </a><a href="https://issues.apache.org/jira/browse/LUCENE-7647" style="text-decoration-line: underline;">https://issues.apache.org/jira/browse/LUCENE-7647</a> <a href="https://github.com/elastic/elasticsearch/pull/22711" style="text-decoration-line: underline;">https://github.com/elastic/elasticsearch/pull/22711 </a>And one issue in X-Pack watcher.These fixes were released in Elastic Stack version 5.2.1.To verify that we had caught all cases, we made a small tool that records when we allocate an Inflater, and when we call end() on it. It is implemented as an agent which instruments the Inflater class. We ran the tool on Elasticsearch 5.1.1, and it showed exactly the places that needed to be fixed, and with 5.2.1 it showed none of them (except a few that are expected due to jar caching). We've seen that when normal memory profiling tools fail to identify a memory issue, it might be outside of the JVM. When this is the case, we need to peek inside it. And since the JVM is just an "ordinary" C++ program, we can use well known tools and techniques for profiling it.

child-leaky-bucket.jpg

child-leaky-bucket-small.jpg

Tracking Down Native Memory Leaks in Elasticsearch

<html><head></head><body><h1>How to make a Dockerfile for Elasticsearch</h1>Docker containers gives you a way to ship and run applications with their environment in an isolated and repeatable way. While there are a myriad of Docker images out there <a href="#fn1" id="ref1">1</a>, creating your own Dockerfile allows you to customize it, for instance by installing plugins, changing the base image, strip out what you don&apos;t need, etc. Dockerfiles also act as a way to document how an application gets installed and deployed.
In this introductory post we will go through how to create a Dockerfile from scratch for running Elasticsearch, and discuss a few things that you need to consider when creating your own.
<h2>Building the image</h2>A Dockerfile is a recipe with steps describing how to build your Docker image. You start from a base image, which gives you the basics needed for&#xA0;running applications, then run steps on top of that, which results in a new image. If you want you can also use the resulting image as a base image for another image. 
The simplest Dockerfile you can create is something like this:
<pre class="prettyprint prettyprinted">FROM ubuntu:14.04
</pre>Put this in a file called&#xA0;<code>Dockerfile</code>.&#xA0; You can now build it by running:
<pre class="prettyprint prettyprinted">docker build -t my-es-image .
</pre>The image has been built and can be run with:
<pre class="prettyprint prettyprinted">docker run --rm -it my-es-image /bin/bash
</pre>While this created an image that is not very useful, we have now learned how to build and test an image. Now let&#x2019;s create a more useful one. Since Elasticsearch requires Java to run, let&#x2019;s install it first.
<pre class="prettyprint prettyprinted">FROM ubuntu:14.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get install -y --no-install-recommends software-properties-common &amp;&amp; add-apt-repository -y ppa:webupd8team/java &amp;&amp; \
 apt-get update &amp;&amp; \
 (echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections) &amp;&amp; \
 apt-get install --no-install-recommends -y oracle-java8-installer &amp;&amp; \
 rm -rf /var/cache/oracle-jdk8-installer &amp;&amp; \
 echo &quot;networkaddress.cache.ttl=60&quot; &gt;&gt; /usr/lib/jvm/java-8-oracle/jre/lib/security/java.security &amp;&amp; \
 apt-get clean &amp;&amp; rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
</pre>This one installs Oracle JDK 8. If you build and run it with the&#xA0;<code>docker run</code>&#xA0;command above then you can test that is works by running <code>java -version</code>.
At this point we&#x2019;re ready to install Elasticsearch. Let&#x2019;s use the apt package.
<pre class="prettyprint prettyprinted">RUN groupadd -g 1000 elasticsearch &amp;&amp; useradd elasticsearch -u 1000 -g 1000

RUN apt-key adv --keyserver pgp.mit.edu --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4 &amp;&amp; \
 add-apt-repository -y &quot;deb http://packages.elastic.co/elasticsearch/2.x/debian stable main&quot; --keyserver https://pgp.mit.edu/ &amp;&amp; \
 apt-get update &amp;&amp; \
 apt-get install -y --no-install-recommends elasticsearch

WORKDIR /usr/share/elasticsearch

RUN set -ex &amp;&amp; for path in data logs config config/scripts; do \
 mkdir -p &quot;$path&quot;; \
 chown -R elasticsearch:elasticsearch &quot;$path&quot;; \
 done
</pre>Before we run it, we should add an elasticsearch.yml file. Create a file named&#xA0;<code>elasticsearch.yml</code> in the same directory as the Dockerfile, with this content:
<pre class="prettyprint prettyprinted">cluster.name: &quot;docker-cluster&quot;
network.host: 0.0.0.0
</pre>Also, to get logging to work with docker we should add a simple&#xA0;<code>logging.yml</code> file:
<pre class="prettyprint prettyprinted">rootLogger: INFO,console
appender:
 console:
 type: console
 layout:
 type: consolePattern
 conversionPattern: &quot;[%d{ISO8601}][%-5p][%-25c] %m%n&quot;
</pre><blockquote>Note: When running with logging to stdout/stderr Docker stores the log in a json file, and it is recommended to specify a max size for the log file to rotate, and a max number of files to keep. E.g.&#xA0;<code>--log-opt max-size=50m --log-opt max-file=10</code> <a href="#fn2" id="ref2">2</a>
</blockquote>To add these files to the container we add the following to the Dockerfile:
<pre class="prettyprint prettyprinted">COPY logging.yml /usr/share/elasticsearch/config/
COPY elasticsearch.yml /usr/share/elasticsearch/config/
</pre>This will bake the files into the image when running&#xA0;<code>docker build</code>. <a href="#fn3" id="ref3">3</a>&#xA0; What&#x2019;s left now is to actually make the container run Elasticsearch at startup.
<pre class="prettyprint prettyprinted">USER elasticsearch

ENV PATH=$PATH:/usr/share/elasticsearch/bin

CMD [&quot;elasticsearch&quot;]

EXPOSE 9200 9300
</pre>To run this, first build it as before, and then run with&#xA0;<code>docker run --rm -p 9200:9200 my-es-image</code>. This opens port 9200 from the container to the host and runs CMD. You should now see the Elasticsearch instance starting.
Note that currently this Elasticsearch instance does not persist the data between runs, since it is ephemeral and we didn&#x2019;t specify any data volumes.
<h2>Persistent storage volumes</h2>Storing data persistently in Docker requires the use of volumes. Volumes are a bit tricky because of the way it works with permissions. By default volumes are using bind-mount, which means that a file belonging to a user with ID 1000 inside the container will be owned by user 1000 on the host, which may or may not be the same actual user.&#xA0; Docker are working on a way of fixing this, and it has been&#xA0;partially implemented, but we need &#x201C;phase 2&#x201D; of user namespaces to solve this fully.
In the meantime, we need some workaround for storing the data. One way to get around it is to hard code the User ID and make sure that it is the same on all machines running that container.&#xA0; If we only care that the data is persisted between restarts, another way is to let the container create the volume so the permissions are correct for the container, since we don&#x2019;t need to access the container contents from the host machine.
Let&#x2019;s do the latter approach&#xA0;that with a feature called &#x201C;named volumes&#x201D;. Create a named volume:
<pre class="prettyprint prettyprinted">docker volume create --name esdata
</pre>You can mount that volume using the&#xA0;<code>-v</code>&#xA0;option:
<pre class="prettyprint prettyprinted">docker run --rm -ti -p 9200:9200 -v esdata:/usr/share/elasticsearch/data my-es-image
</pre>Even if you restart your Elasticsearch container now it should preserve all data.
<blockquote>Note: This requires at least version 1.10 of Docker
</blockquote><h2>Memory and heap size</h2>By default memory for a container is unbounded. If you want to limit for the max memory the container uses you can specify e.g.&#xA0;<code>--memory=&quot;4g&quot;</code> with <code>docker run</code>. You should also set the heap size for Elasticsearch. The normal recommendation with allocating half of the memory to the heap also applies. For example:
<pre class="prettyprint prettyprinted">docker run --rm -ti -p 9200:9200 -v esdata:/usr/share/elasticsearch/data --memory=&quot;4g&quot; -e ES_HEAP_SIZE=2g my-es-image
</pre><h2>Conclusion</h2>We have created a basic docker image which runs Elasticsearch and stores&#xA0;the data persistently. It is basic, but it&apos;s a starting point. Next steps could be to <a href="https://www.elastic.co/blog/docker-networking">set up networking</a> and <a href="https://www.elastic.co/blog/elasticsearch-docker-plugin-management">install plugins</a>, which we will look at in follow-up blog posts.
 
<hr><div class="footnotes">
	<ol>
		<li id="fn1">The Elasticsearch image on Docker hub is created by Docker, and not yet officially supported by Elastic.<a href="#ref1" title="Return.">&#x21A9;</a></li>
		<li id="fn2">You might also consider using another <a href="https://docs.docker.com/engine/admin/logging/overview/">logging driver</a> than the json-file if performance is an issue.<a href="#ref2" title="Return.">&#x21A9;</a></li>
		<li id="fn3">An alternative to copying the configuration files into the image at build time is to mount the the config directory as a volume. This also allows you to <a href="http://elasticcompute.io/2016/01/21/runtime-secrets-with-docker-containers/">avoid putting secrets</a> into the image.<a href="#ref3" title="Return.">&#x21A9;</a></li>
	</ol>
	
</div></body></html>

How to make a Dockerfile for Elasticsearch

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-henrik-nordvik-300x300.jpg

Principal Software Engineer I

Henrik Nordvik

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Henrik Nordvik

Principal Software Engineer I, Elastic

Tracking Down Native Memory Leaks in Elasticsearch

How to make a Dockerfile for Elasticsearch