Our Cloud service experienced issues with managing memory for the clusters.  This blog is a postmortem of the incident.

Elastic Cloud is on the tail end of eliminating a mix of memory issues that has caused problems for a lot of low-memory nodes, and in some rare cases even large nodes. Following the memory problems, we experienced connectivity issues on a handful of servers in eu-west-1 that affected any cluster with at least one node on these impacted servers. After investigation, we found that there were a number of small problems that, when combined, created larger issues. &nbsp;This post attempts to summarise the breadth of things that all contributed to the problems, and the extent of testing we’re ramping up to avoid repeating our mistakes. You might learn a thing or two about Linux, Docker, memory accounting, glibc, JVM-settings, Netty, and Elasticsearch as well. We sure have. Scope&nbsp;of the memory problemElastic Cloud runs tens of thousands of Elasticsearch nodes. These nodes run on servers with memory ranging from 30-244 GiB and 4-32 cores, which we call our “allocator pool”. A single server can host a lot of Elasticsearch nodes, each running in containers getting a reserved slice of memory and a (boostable) slice of the CPU. The servers run a small number of different kernel and Docker versions, and we’ll get back to why. The memory issues can be categorised as:<ul><li dir="ltr">High memory pressure from Elasticsearch causing increased GC-load and latencies, and eventually the JVM running out of heap space. There were several things that could lead to this.</li><li dir="ltr">Growth in the JVM’s non-heap memory, eating away memory intended for page cache and possibly causing kernel-level OOM-reaping.</li><li dir="ltr">Because we used long-running containers, kernel memory was being incorrectly accounted, causing the kernel to kill processes in a container almost as soon as the node started.</li></ul>All of these could affect a single node, but just one would be sufficient to make the node unreliable. Whenever a process is restarted due to memory issues, we log the event and notify the account owner and operational contacts (rate limiting to max one email per 6 hours). While Elasticsearch keeps getting more careful against running out of memory, it’s not uncommon for an overloaded cluster to run out of memory. Thus, we know that in the period we had the most issues, approximately 1% of the running clusters were affected. That’s a lot of clusters, but as most mitigations affect every cluster and some required restarting Elasticsearch and/or upgrading, we needed to proceed carefully to not cause any problems to the majority of nodes not having any issues. On environment variety Elastic Cloud is based on the acquisition of Found, which launched a hosted Elasticsearch service in 2012. Having managed lots of containers since before Docker even existed and container schedulers were buzzwords, we have a lot of experience in how container-features can cause Linux to crash, or sometimes worse, cause nodes to slow down to a crawl. Even the 4.4 kernel series in Ubuntu LTS recently had OOM-issues. With the exception of security patches, we’re typically very slow when it comes to upgrading Linux and Docker: issues with these components can severely hurt our reliability or create significant ops workload &nbsp;to clean up containers that are not being created or destroyed correctly. Docker is a fast-moving technology, and generally only the most recent versions receive security patches. This makes it important to keep up, and we were gradually increasing the numbers of servers running more recent versions as we gained confidence in it. Our server fleet is also composed of servers of varying size. Smaller servers limit the blast radius if there’s an issue, while larger servers are necessary to host the beefier clusters. Depending on available capacity during provisioning, a small 1GB node can end up on a massive server. A node will be allotted the same CPU time regardless of the numbers of cores available, so performance differences are small between servers. There are settings that rely &nbsp;on the core count, however, and we didn’t properly cover all the bases of settings that look at cores. This could pose problems for a small node landing on a large server.Having run the 3.19-series of the kernel for a long time without issues, it took some time before we suspected it could be the issue. This will be described more later, but we’ve found that &nbsp;Docker ≥1.12 has problems on Linux &lt;4.4. To start with, we turned every stone related to Elasticsearch and the JVM. Elasticsearch, Lucene, and the JVMElasticsearch needs both heap space and page cache to perform well. A cluster with 1 GB memory on Cloud gets a little less than half the memory for heap space, to leave memory for page cache and non-heap JVM usage. There were a few issues in the early 5.0s that could cause a small node to quickly OOM as segments grew large enough to consume the available buffer space the S3 snapshotter could use, which was changed to 100 MB. This would be about 20% of a 1GB nodes available space, quickly leading to issues. That was quickly identified and remedied and every cluster was upgraded to apply the fix. However, we still saw Elasticsearch 5.x use a lot more non-heap memory than 2.x, which we eventually attributed to Elasticsearch 5 upgrading to Netty 4. Disabling Netty’s pooled allocator and recycler further reduced non-heap memory. That still wasn’t enough, some nodes kept on OOM-ing – but now by the kernel’s OOM-reaper, which triggers if a process in a container with limited memory exceeds its memory. Increased non-heap usage would normally result in performance reductions, and not processes getting killed by the kernel OOM-reaper. So we knew we still had an issue. We found more tweaks that improved the memory usage issues:<ul><li dir="ltr">Upgrading to JVM 8 turned tiered compilation on by default, something not really necessary for Elasticsearch. This would eat up 48MB memory for additional code caches.</li><li dir="ltr">glibc’s memory allocator could waste a lot of memory on servers with many cores. A colleague coming in from Prelert has described the interactions of the <a href="http://info.prelert.com/blog/java-8-and-virtual-memory-on-linux">JVM and virtual memory on Linux</a> as they relate to that change, which could waste a lot of memory for a small node running on a large server.</li><li dir="ltr">There were a number of small fixes in Elasticsearch between 5.0.0 and 5.2.2 that helped with memory usage. &nbsp;&nbsp;For example, not <a href="https://github.com/elastic/elasticsearch/pull/22711">closing a stream</a> could leak memory.</li><li dir="ltr">We reduced the number of JVM allocated GC threads. &nbsp;</li></ul>We are also expanding our test suites to include tests that specifically address long running containers under load, and measure memory usage. Kernel and Docker bugsAfter much debugging, we found that specific combinations of kernel version and Docker version create a major problem with memory accounting. &nbsp;In our case, combining kernel version 3.19 with Docker version 1.12 exposed this bug. &nbsp;We had been running the 3.19 kernel for a long time, and it wasn’t immediately obvious that the kernel was a contributing factor to memory issues. The core of the issue is that Docker 1.12 turns on kmem accounting. &nbsp;In 3.x versions of the Linux kernel, this causes problems because of a <a href="https://lwn.net/Articles/628829/">slab shrinker issue</a>. &nbsp;In a nutshell, this causes the kernel to think that there is more kernel memory used than there actually is, and it starts killing processes to reclaim memory. &nbsp;Eventually it kills the JVM, which obviously hurts the cluster. &nbsp;There is a fix for the slab shrinker issue in kernel versions &gt;= 4.0. &nbsp;Our testing led us to combine kernel version 4.4 with Docker 1.12. &nbsp;This combination solved the kmem accounting problems. Looking forwardAs you can see, there were a number of issues that combined to create a kind of “perfect storm” of memory issues. &nbsp;We are now at a point where we’re convinced we’ve identified all of the major issues and are a long way toward addressing them throughout our fleet. The total number of affected clusters in our SaaS environment was around 1%. &nbsp;While this seems like a small number, we’re committed to reaching out to affected customers and offering explanations and help. &nbsp;Although this issue affected clusters of all sizes, smaller clusters were the fastest to be affected due to the already limited amount of memory. &nbsp;Since trial customers tend to run smaller clusters, we’ll be contacting trial customers who were active during the affected time period and offering new or extended trials.

banner-the-thinker-blue.jpg

thumb-the-thinker-blue.jpg

Memory Issues We'll Remember

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Chris Overton

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Chris Overton

Memory Issues We'll Remember

Sign up for Elastic Cloud free trial