<html><head></head><body>On Monday, February 4, 2019, at roughly 02:50 UTC, Elastic Cloud customers with deployments in the AWS us-east-1 region&#xA0;experienced degraded access to their clusters.
After any incident, we perform a detailed post mortem to understand what went wrong, find the root causes, and discover what we can do to ensure that it doesn&#x2019;t happen again.
We&#x2019;d like to share the root cause analysis we performed, the actions we&#x2019;ve taken, and those we intend to take in the future to prevent incidents like this from happening again.
If you have additional concerns or questions, don&#x2019;t hesitate to reach out to us through support@elastic.co.
Sincerely,
The Elastic Cloud team
<h2>Background </h2>At a high level, the Elastic Cloud backend in each supported region consists of three main layers:
<ul>
	<li>Allocator layer: Hosts for Elasticsearch and Kibana instances.</li>
	<li>Coordination layer: This layer holds the system state, connected allocators, and maintains the actual location of each cluster&apos;s nodes and Kibana instances. This is implemented by a three-node Apache <a href="https://zookeeper.apache.org/">ZooKeeper</a> ensemble.</li>
	<li>Proxy layer with two parts:
	<ul>
		<li>Proxy/routing layer: Routes traffic to cluster nodes and Kibana instances based on that cluster&#x2019;s virtual host name, fronted by a load-balancing layer which load balances between multiple proxy instances. </li>
	</ul>
	<ul>
		<li>Proxy <a href="https://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperObservers.html">ZooKeeper Observer</a> layer: This is a replica of the coordination layer and is used by the proxy services to reduce load on the coordination layer during normal operation.</li>
	</ul></li>
</ul>To enable inter-host traffic using TLS, an ancillary container is used for services like Kibana to communicate internally to its Elasticsearch cluster securely. Each allocator host is running that container to allow secure, auditable connections to other hosts within Elastic Cloud.
<h2>What Happened</h2>On 2019-02-04 Elastic Cloud customers experienced cluster connectivity instability issues and later a major service disruption in the AWS us-east-1 region. Deployments of Elasticsearch Service on&#xA0;Elastic Cloud in the AWS&#xA0;us-east-1 region would have been partially or completely unavailable between 02:50 and 09:28 UTC.
The majority of customers in us-east-1 also experienced Kibana access disruptions from 02:50 to 09:28 UTC, and a smaller number of customers had degraded access to&#xA0;their Kibana instances until 18:44 UTC.
The Elastic Cloud User Console was in a degraded state from 02:50 to 07:17 UTC and all customers using the Elastic Cloud Console would have seen increased timeouts as they were trying to view their active deployments.
The incident began following a routine patching procedure for the coordination layer in our AWS us-east-1 region. Despite following the same process as the other 12 regions, patching of our coordination service, ZooKeeper, in this region did not go as expected, resulting in an outage of the coordination services.
All times below are in UTC.
<ul>
	<li>2019-02-04 02:25-02:45: First and second ensemble members were replaced with patched hosts.</li>
	<li>2019-02-04 02:45: The ZooKeeper ensemble came under high load (CPU, IO, memory), with large numbers of client connection failures and re-connects across the ensemble. This continued for several minutes. The ensemble then appeared to stabilize, machine load decreased, outstanding request counts decreased, and connection failures decreased. </li>
	<li>2019-02-04 02:50: The ZooKeeper ensemble undertook a leader election. Logging indicated that the ensemble leader was overwhelmed by incoming connections and shut down. The ensemble wasn&#x2019;t able to quickly re-establish and maintain a quorum due to the incoming request load. </li>
	<li>2019-02-04 02:50: An increase in HTTP 5xx errors was identified on the load balancer.</li>
	<li>2019-02-04 02:55: Latency within the ZooKeeper ensemble increased due to ensemble instability, which in turn led to further client reconnections and instability in the ZooKeeper observer layer.</li>
	<li>2019-02-04 03:22: Sweeping restart of internal services to reduce client connection problems.</li>
	<li>2019-02-04 03:31: Quorum stabilized and engineering monitored the situation for ~20 minutes and, separately, a discussion was had about next steps. A call was made to roll forward with the operation.</li>
	<li>2019-02-04 04:07: Roll-forward began for the remaining ensemble member.</li>
	<li>2019-02-04 04:15: System-level unresponsiveness was observed on the final ensemble member; troubleshooting began, but multiple issues cropped up.</li>
	<li>2019-02-04 04:39: Isolated ensemble by cutting off client connections to stabilize the system.</li>
	<li>2019-02-04 04:46: Roll-back&#xA0;of the remaining ensemble member began after host instability was observed.</li>
	<li>2019-02-04 05:25: Roll-back&#xA0;of the second ensemble member began due to host instability.</li>
	<li>2019-02-04 05:48: First-seen runc bugs led to CPU softlocks and rendered the system unresponsive for two members of the ZooKeeper ensemble. The hosts had to be rebooted to mitigate this bug.</li>
	<li>2019-02-04 06:00: Sweeping restart of internal services to reduce client connection problems; started working to identify what exactly was causing the host instability within ensemble members.</li>
	<li>2019-02-04 06:45: Identified the root issue causing ensemble member instability: when entering the network namespace of a TLS proxying service container, runc would trigger a kernel OOPS and CPU softocks spanning minutes.</li>
	<li>2019-02-04 07:10: A stable quorum was established by reducing client load and reducing operator debugging actions on the impacted ZooKeeper hosts.</li>
	<li>2019-02-04 07:17: Incoming traffic was reintroduced and the customer control plane was brought back online. </li>
	<li>2019-02-04 07:30: Once the client connections were re-established, the proxy layer began to exhibit high load due to a large influx of external re-connections. This caused instability in the proxy ZooKeeper observer layer, which led to continued unavailability in the proxy layer itself (i.e., customers still encountered issues connecting to their clusters). </li>
	<li>2019-02-04 09:19: Once the ZooKeeper ensemble and observers were stable, the proxy fleet was scaled up and out to handle the increased traffic to ES clusters. A three-fold increase of requests to our proxy layer was noted.</li>
	<li>2019-02-04 09:28: Availability to Elasticsearch deployments was restored. All proxies were now serving traffic and the error rate dropped to pre-incident levels.</li>
</ul>While availability to Elasticsearch deployments was fully restored at 2019-02-04 09:28, Kibana deployments suffered an extended period of unavailability:
<ul>
	<li>2019-02-04 09:35: CPU usage on Kibana host instances was observed to be much higher than usual. It was identified that our internal TLS proxying service was the main contributor.</li>
	<li>2019-02-04 09:53: Kibana instances were not able to connect to their target Elasticsearch clusters due the internal TLS proxying service being unable to forward requests to the proxy layer. As Kibana was not closing sockets after a failed attempt, the number of open sockets grew out of bounds, filling the nf_conn_track table and causing a high strain on the internal proxying containers. </li>
	<li>2019-02-04 12:30: A remediation runbook was developed and applied to mitigate the issue with Kibana instances exhibiting the connection issue. The runbook restarted the internal proxying container and Kibana instances, and also applied sysctl limits to prevent these issues from recurring in the future. </li>
	<li>2019-02-04 18:44: Kibana access was fully restored.</li>
</ul><h2>Root Cause</h2>At a high level, the problems included a combination of the below:
<ol>
	<li>Failure in coordination layer</li>
	<li>Failure in proxy/routing layer</li>
</ol>While performing routine maintenance of the Elasticsearch Service coordination layer in the AWS us-east-1 region, an engineer carrying out the operation encountered an unfamiliar error. The engineer was following the documented procedure of replacing the underlying hosts, which had previously been successful when performed in other regions. Service metrics had reported the hosts as healthy, thus signaling that it was safe to proceed with the maintenance; however, the metrics proved to be insufficient in conveying the state of individual hosts and of the coordination layer as a whole. This led to unanticipated instability on and between hosts, which ultimately led to an overall failure of the coordination layer.
Due to dependencies of other platform components with the coordination layer, instability and loss of quorum in the coordination layer caused a major service disruption. A set of unrelated issues were encountered while investigating the primary source of the outage, which contributed to a further delay of the final resolution. Individually successful mitigation actions by our response team proved to be temporarily inefficient due to the scale of the failures being addressed, making a full resolution both more complicated and more difficult to achieve.
<h2>Resolution</h2>Quorum within the ZooKeeper ensemble was re-established by removing client load where possible, giving the ensemble time to stabilize before gradually re-introducing client load.
The ZooKeeper observer layer utilized by the proxies was stabilized by increasing the <code>initLimit</code> setting. This setting controls the amount of time followers are allowed to connect and sync to a leader. Increasing this limit stabilized the observer layer, which flowed down to the proxy layer.
The extended availability issues with Kibana instances were resolved by restarting the internal proxying container and affected Kibana instances en masse. As part of the RCA investigation, it was identified that a set of previously unknown bugs in Kibana was resulting in connection leaks and HTTP request amplification back to the load balancing layer, driving extra traffic to the proxy layer and Elasticsearch deployments.
<h2>Impact</h2><h3>Coordination layer </h3><ul>
	<li>100% of customers in us-east-1</li>
	<li>Inability to create, modify, or delete clusters through the customer control panel</li>
	<li>ZooKeeper was unavailable between 02:50 UTC and 07:17 UTC</li>
</ul><h3>Proxy layer </h3><ul>
	<li>Proxy layer was impacted between 02:50 UTC and 09:15 UTC and 100% of customers within us-east-1 were affected in some way; our data indicates that every Elasticsearch cluster saw some failures during this time period</li>
	<li>Inability to access Elasticsearch/Kibana/APM services within running clusters</li>
	<li>Proxy remediation work began when ZooKeeper was made available again at 07:30 UTC</li>
</ul><h3>Kibana </h3>This was a result of a set of cascading failures. Access to Kibana was impacted by the aforementioned proxy layer outage.
<ul>
	<li>Kibana instances were impacted between 02:50 UTC and 18:44 UTC</li>
	<li>Access was initially impacted by proxy layer availability as well as service discovery issues </li>
	<li>A number of hosts affected by a bug in our internal TLS proxying service triggered by extended unavailability were identified and fixed</li>
	<li>A socket leak in Kibana was identified and mitigated by restarting Kibana containers</li>
	<li>Other issues with Kibana that were determined to lead to request amplification are also being investigated</li>
</ul><h2>Action Items</h2>A number of action items have come from this extended incident and are actively being worked on by the Elastic Cloud team. These include the following:
<h3>Engineering and Architecture
</h3><ul>
	<li>Reduced the dataset size carried by our ZooKeeper ensembles by over 60%. By doing so, network and disk I/O requirements for ZooKeeper are greatly reduced during normal operation, as well as providing faster recovery times for ensemble members joining the quorum. (Completed)
	</li>
	<li>Reduced the number of health-checks and intervals required to allow a proxy node, marked as healthy, to join our load balancing layer as eligible targets to route traffic. This action item aims to reduce the mean time to recovery. (Completed)
	</li>
	<li>Further revamp the logic around our proxy health-checks to not account for ZooKeeper connectivity as part of the checklist. With this change, once a proxy is successfully initialized, and has received its configuration and cluster routes, it will no longer drift to an unhealthy state due to the state of the supporting ZooKeeper ensemble. This change aims to sacrifice a potential small consistency skew over general availability. (In progress)
	</li>
	<li>Canarying the ground up rewrite of our proxy layer in production regions, as one of the final steps to switch to our proxy v2 architecture. The v2 architecture decouples the proxy service from the ZooKeeper state. (In progress)
	</li>
	<li>Improve visibility around ZooKeeper by further investing in our internal logging strategy and introducing better metrics around the ZooKeeper processes, the JVM, and container/host systems. (In progress)
	</li>
	<li>Improve resiliency in Kibana routes discovery methods. (In progress)
	</li>
	<li>Identifying and solving issues with Kibana that lead to internal HTTP request amplifications during availability events. (In progress)
	</li>
	<li>Identifying and solving issues with Kibana that lead to connection leaking. (In progress)
	</li>
	<li>Identified and mitigated issues with internal monitoring agents not employing proper backoff strategies with Jitter. (Completed)
	</li>
</ul><h3>Process and Communications
</h3><ul>
	<li>Formalized our maintenance matrix to reduce exposure of critical services within a region. We will be operating within specific timeframes for each region to limit the exposure of a failure to non-critical times. (Completed)
	</li>
	<li>Provide our team with an improved decision criteria matrix in runbooks that involves regular maintenance operations with the coordination layer. (In progress)
	</li>
	<li>Established a feedback loop with our support leadership team for communications that go out through support or the status page. This will make sure we are answering the right questions and providing relevant updates. (Completed)</li>
</ul>If any additional follow-up actions for this incident are decided upon, we will update this post accordingly. We thank our customers for their patience and understanding and look forward to fully implementing the necessary measures to prevent issues like this in the future.</body></html>

blog-banner-release-elasticsearch.png

blog-thumb-release-elasticsearch.png

Elastic Cloud Incident Report: February 4, 2019

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Ben Osborne

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Ben Osborne

Elastic Cloud Incident Report: February 4, 2019

Sign up for Elastic Cloud free trial