<html><head></head><body>On Monday, February 4, 2019, at roughly 02:50 UTC, Elastic Cloud customers with deployments in the AWS us-east-1 region&#xA0;experienced degraded access to their clusters.
After any incident, we perform a detailed post mortem to understand what went wrong, find the root causes, and discover what we can do to ensure that it doesn&#x2019;t happen again.
We&#x2019;d like to share the root cause analysis we performed, the actions we&#x2019;ve taken, and those we intend to take in the future to prevent incidents like this from happening again.
If you have additional concerns or questions, don&#x2019;t hesitate to reach out to us through support@elastic.co.
Sincerely,
The Elastic Cloud team
<h2>Background </h2>At a high level, the Elastic Cloud backend in each supported region consists of three main layers:
<ul>
	<li>Allocator layer: Hosts for Elasticsearch and Kibana instances.</li>
	<li>Coordination layer: This layer holds the system state, connected allocators, and maintains the actual location of each cluster&apos;s nodes and Kibana instances. This is implemented by a three-node Apache <a href="https://zookeeper.apache.org/">ZooKeeper</a> ensemble.</li>
	<li>Proxy layer with two parts:
	<ul>
		<li>Proxy/routing layer: Routes traffic to cluster nodes and Kibana instances based on that cluster&#x2019;s virtual host name, fronted by a load-balancing layer which load balances between multiple proxy instances. </li>
	</ul>
	<ul>
		<li>Proxy <a href="https://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperObservers.html">ZooKeeper Observer</a> layer: This is a replica of the coordination layer and is used by the proxy services to reduce load on the coordination layer during normal operation.</li>
	</ul></li>
</ul>To enable inter-host traffic using TLS, an ancillary container is used for services like Kibana to communicate internally to its Elasticsearch cluster securely. Each allocator host is running that container to allow secure, auditable connections to other hosts within Elastic Cloud.
<h2>What Happened</h2>On 2019-02-04 Elastic Cloud customers experienced cluster connectivity instability issues and later a major service disruption in the AWS us-east-1 region. Deployments of Elasticsearch Service on&#xA0;Elastic Cloud in the AWS&#xA0;us-east-1 region would have been partially or completely unavailable between 02:50 and 09:28 UTC.
The majority of customers in us-east-1 also experienced Kibana access disruptions from 02:50 to 09:28 UTC, and a smaller number of customers had degraded access to&#xA0;their Kibana instances until 18:44 UTC.
The Elastic Cloud User Console was in a degraded state from 02:50 to 07:17 UTC and all customers using the Elastic Cloud Console would have seen increased timeouts as they were trying to view their active deployments.
The incident began following a routine patching procedure for the coordination layer in our AWS us-east-1 region. Despite following the same process as the other 12 regions, patching of our coordination service, ZooKeeper, in this region did not go as expected, resulting in an outage of the coordination services.
All times below are in UTC.
<ul>
	<li>2019-02-04 02:25-02:45: First and second ensemble members were replaced with patched hosts.</li>
	<li>2019-02-04 02:45: The ZooKeeper ensemble came under high load (CPU, IO, memory), with large numbers of client connection failures and re-connects across the ensemble. This continued for several minutes. The ensemble then appeared to stabilize, machine load decreased, outstanding request counts decreased, and connection failures decreased. </li>
	<li>2019-02-04 02:50: The ZooKeeper ensemble undertook a leader election. Logging indicated that the ensemble leader was overwhelmed by incoming connections and shut down. The ensemble wasn&#x2019;t able to quickly re-establish and maintain a quorum due to the incoming request load. </li>
	<li>2019-02-04 02:50: An increase in HTTP 5xx errors was identified on the load balancer.</li>
	<li>2019-02-04 02:55: Latency within the ZooKeeper ensemble increased due to ensemble instability, which in turn led to further client reconnections and instability in the ZooKeeper observer layer.</li>
	<li>2019-02-04 03:22: Sweeping restart of internal services to reduce client connection problems.</li>
	<li>2019-02-04 03:31: Quorum stabilized and engineering monitored the situation for ~20 minutes and, separately, a discussion was had about next steps. A call was made to roll forward with the operation.</li>
	<li>2019-02-04 04:07: Roll-forward began for the remaining ensemble member.</li>
	<li>2019-02-04 04:15: System-level unresponsiveness was observed on the final ensemble member; troubleshooting began, but multiple issues cropped up.</li>
	<li>2019-02-04 04:39: Isolated ensemble by cutting off client connections to stabilize the system.</li>
	<li>2019-02-04 04:46: Roll-back&#xA0;of the remaining ensemble member began after host instability was observed.</li>
	<li>2019-02-04 05:25: Roll-back&#xA0;of the second ensemble member began due to host instability.</li>
	<li>2019-02-04 05:48: First-seen runc bugs led to CPU softlocks and rendered the system unresponsive for two members of the ZooKeeper ensemble. The hosts had to be rebooted to mitigate this bug.</li>
	<li>2019-02-04 06:00: Sweeping restart of internal services to reduce client connection problems; started working to identify what exactly was causing the host instability within ensemble members.</li>
	<li>2019-02-04 06:45: Identified the root issue causing ensemble member instability: when entering the network namespace of a TLS proxying service container, runc would trigger a kernel OOPS and CPU softocks spanning minutes.</li>
	<li>2019-02-04 07:10: A stable quorum was established by reducing client load and reducing operator debugging actions on the impacted ZooKeeper hosts.</li>
	<li>2019-02-04 07:17: Incoming traffic was reintroduced and the customer control plane was brought back online. </li>
	<li>2019-02-04 07:30: Once the client connections were re-established, the proxy layer began to exhibit high load due to a large influx of external re-connections. This caused instability in the proxy ZooKeeper observer layer, which led to continued unavailability in the proxy layer itself (i.e., customers still encountered issues connecting to their clusters). </li>
	<li>2019-02-04 09:19: Once the ZooKeeper ensemble and observers were stable, the proxy fleet was scaled up and out to handle the increased traffic to ES clusters. A three-fold increase of requests to our proxy layer was noted.</li>
	<li>2019-02-04 09:28: Availability to Elasticsearch deployments was restored. All proxies were now serving traffic and the error rate dropped to pre-incident levels.</li>
</ul>While availability to Elasticsearch deployments was fully restored at 2019-02-04 09:28, Kibana deployments suffered an extended period of unavailability:
<ul>
	<li>2019-02-04 09:35: CPU usage on Kibana host instances was observed to be much higher than usual. It was identified that our internal TLS proxying service was the main contributor.</li>
	<li>2019-02-04 09:53: Kibana instances were not able to connect to their target Elasticsearch clusters due the internal TLS proxying service being unable to forward requests to the proxy layer. As Kibana was not closing sockets after a failed attempt, the number of open sockets grew out of bounds, filling the nf_conn_track table and causing a high strain on the internal proxying containers. </li>
	<li>2019-02-04 12:30: A remediation runbook was developed and applied to mitigate the issue with Kibana instances exhibiting the connection issue. The runbook restarted the internal proxying container and Kibana instances, and also applied sysctl limits to prevent these issues from recurring in the future. </li>
	<li>2019-02-04 18:44: Kibana access was fully restored.</li>
</ul><h2>Root Cause</h2>At a high level, the problems included a combination of the below:
<ol>
	<li>Failure in coordination layer</li>
	<li>Failure in proxy/routing layer</li>
</ol>While performing routine maintenance of the Elasticsearch Service coordination layer in the AWS us-east-1 region, an engineer carrying out the operation encountered an unfamiliar error. The engineer was following the documented procedure of replacing the underlying hosts, which had previously been successful when performed in other regions. Service metrics had reported the hosts as healthy, thus signaling that it was safe to proceed with the maintenance; however, the metrics proved to be insufficient in conveying the state of individual hosts and of the coordination layer as a whole. This led to unanticipated instability on and between hosts, which ultimately led to an overall failure of the coordination layer.
Due to dependencies of other platform components with the coordination layer, instability and loss of quorum in the coordination layer caused a major service disruption. A set of unrelated issues were encountered while investigating the primary source of the outage, which contributed to a further delay of the final resolution. Individually successful mitigation actions by our response team proved to be temporarily inefficient due to the scale of the failures being addressed, making a full resolution both more complicated and more difficult to achieve.
<h2>Resolution</h2>Quorum within the ZooKeeper ensemble was re-established by removing client load where possible, giving the ensemble time to stabilize before gradually re-introducing client load.
The ZooKeeper observer layer utilized by the proxies was stabilized by increasing the <code>initLimit</code> setting. This setting controls the amount of time followers are allowed to connect and sync to a leader. Increasing this limit stabilized the observer layer, which flowed down to the proxy layer.
The extended availability issues with Kibana instances were resolved by restarting the internal proxying container and affected Kibana instances en masse. As part of the RCA investigation, it was identified that a set of previously unknown bugs in Kibana was resulting in connection leaks and HTTP request amplification back to the load balancing layer, driving extra traffic to the proxy layer and Elasticsearch deployments.
<h2>Impact</h2><h3>Coordination layer </h3><ul>
	<li>100% of customers in us-east-1</li>
	<li>Inability to create, modify, or delete clusters through the customer control panel</li>
	<li>ZooKeeper was unavailable between 02:50 UTC and 07:17 UTC</li>
</ul><h3>Proxy layer </h3><ul>
	<li>Proxy layer was impacted between 02:50 UTC and 09:15 UTC and 100% of customers within us-east-1 were affected in some way; our data indicates that every Elasticsearch cluster saw some failures during this time period</li>
	<li>Inability to access Elasticsearch/Kibana/APM services within running clusters</li>
	<li>Proxy remediation work began when ZooKeeper was made available again at 07:30 UTC</li>
</ul><h3>Kibana </h3>This was a result of a set of cascading failures. Access to Kibana was impacted by the aforementioned proxy layer outage.
<ul>
	<li>Kibana instances were impacted between 02:50 UTC and 18:44 UTC</li>
	<li>Access was initially impacted by proxy layer availability as well as service discovery issues </li>
	<li>A number of hosts affected by a bug in our internal TLS proxying service triggered by extended unavailability were identified and fixed</li>
	<li>A socket leak in Kibana was identified and mitigated by restarting Kibana containers</li>
	<li>Other issues with Kibana that were determined to lead to request amplification are also being investigated</li>
</ul><h2>Action Items</h2>A number of action items have come from this extended incident and are actively being worked on by the Elastic Cloud team. These include the following:
<h3>Engineering and Architecture
</h3><ul>
	<li>Reduced the dataset size carried by our ZooKeeper ensembles by over 60%. By doing so, network and disk I/O requirements for ZooKeeper are greatly reduced during normal operation, as well as providing faster recovery times for ensemble members joining the quorum. (Completed)
	</li>
	<li>Reduced the number of health-checks and intervals required to allow a proxy node, marked as healthy, to join our load balancing layer as eligible targets to route traffic. This action item aims to reduce the mean time to recovery. (Completed)
	</li>
	<li>Further revamp the logic around our proxy health-checks to not account for ZooKeeper connectivity as part of the checklist. With this change, once a proxy is successfully initialized, and has received its configuration and cluster routes, it will no longer drift to an unhealthy state due to the state of the supporting ZooKeeper ensemble. This change aims to sacrifice a potential small consistency skew over general availability. (In progress)
	</li>
	<li>Canarying the ground up rewrite of our proxy layer in production regions, as one of the final steps to switch to our proxy v2 architecture. The v2 architecture decouples the proxy service from the ZooKeeper state. (In progress)
	</li>
	<li>Improve visibility around ZooKeeper by further investing in our internal logging strategy and introducing better metrics around the ZooKeeper processes, the JVM, and container/host systems. (In progress)
	</li>
	<li>Improve resiliency in Kibana routes discovery methods. (In progress)
	</li>
	<li>Identifying and solving issues with Kibana that lead to internal HTTP request amplifications during availability events. (In progress)
	</li>
	<li>Identifying and solving issues with Kibana that lead to connection leaking. (In progress)
	</li>
	<li>Identified and mitigated issues with internal monitoring agents not employing proper backoff strategies with Jitter. (Completed)
	</li>
</ul><h3>Process and Communications
</h3><ul>
	<li>Formalized our maintenance matrix to reduce exposure of critical services within a region. We will be operating within specific timeframes for each region to limit the exposure of a failure to non-critical times. (Completed)
	</li>
	<li>Provide our team with an improved decision criteria matrix in runbooks that involves regular maintenance operations with the coordination layer. (In progress)
	</li>
	<li>Established a feedback loop with our support leadership team for communications that go out through support or the status page. This will make sure we are answering the right questions and providing relevant updates. (Completed)</li>
</ul>If any additional follow-up actions for this incident are decided upon, we will update this post accordingly. We thank our customers for their patience and understanding and look forward to fully implementing the necessary measures to prevent issues like this in the future.</body></html>

blog-banner-release-elasticsearch.png

blog-thumb-release-elasticsearch.png

Elastic Cloud Incident Report: February 4, 2019

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-ben-osborne-300x300.jpg

Site Reliability Engineer Team Lead

Ben Osborne

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Ben Osborne

Site Reliability Engineer Team Lead, Elastic

Elastic Cloud Incident Report: February 4, 2019