New Elastic APM integration for Elastic Enterprise Search enables our developers to quickly pinpoint performance bottlenecks in Enterprise Search.

blog-elastic-apm-1.png

blog-elastic-apm-2.png

blog-elastic-apm-3.png

blog-elastic-apm-4.png

blog-elastic-apm-5.png

blog-elastic-apm-6.png

blog-elastic-apm-7.png

blog-elastic-apm-8.png

blog-elastic-services.png

blog-elastic-apm-9.png

blog-elastic-apm-10.png

blog-elastic-apm-11.png

blog-elastic-apm-12.png

blog-thumb-search-results-orange.png

Monitoring Elastic Enterprise Search performance using Elastic APM

This follow-up blog post walks you through the exciting reporting improvements in X-Pack and provides tips on troubleshooting a few common issues.

Back in October 2016, I wrote a blog post (<a href="https://www.elastic.co/blog/you_get_a_report_you_get_a_report">You get a report! You get a report!</a>) on how to get started with the then-newly released X-Pack reporting <a href="https://www.elastic.co/products/x-pack/reporting">feature</a>. A lot has changed in this feature since its original release, and a quickly growing number of users are adopting it as the primary way to share reports of their favorite Kibana visualizations, dashboards and saved searches. New to Kibana or just not on the latest version? Check out our <a href="https://www.elastic.co/guide/en/kibana/current/reporting-getting-started.html">documentation</a> for the latest getting started information.
Over the past year, we've taken your feedback on our reporting feature and turned it into a reality. There are a number of new enhancements available in reporting today, requested by you — our users. This follow-up blog post walks you through these exciting improvements and provides tips on <a href="#issues">troubleshooting a few common issues</a>.
<h2>Recent Improvements and Enhancements</h2><h3>You get a CSV export!</h3>The ability to export raw documents from Discover (<a href="https://www.elastic.co/guide/en/kibana/6.x/save-open-search.html">saved search</a>), one of the most requested enhancement requests, is available in the Basic (free) version of X-Pack 6.0 and above. Prior to 6.0, saved searches were only captured as screenshots within the PDF report. With CSV export, raw documents from saved searches are now exported using the popular CSV format which makes it easy to import into Excel and other integration tools. To export raw documents from a saved search on the Discover screen, use the new CSV export feature (Discover &gt; Reporting &gt; Generate CSV).
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt123793e1908bae23/5c306936ebb7714d660270f8/csv_export.png" data-sys-asset-uid="blt123793e1908bae23" alt="csv_export.png">
With a <a href="https://www.elastic.co/guide/en/x-pack/6.2/actions-email.html#email-action-reports">Watcher</a> integration, you can <a href="https://www.elastic.co/guide/en/kibana/6.2/automating-report-generation.html">automate</a> the sending of CSV export of saved searches to supplement the PDF report with raw documents. The following is an example watch that attaches both the PDF report and a CSV export of the saved search in the same email. It is much easier with the reporting email attachment type available in versions 5.1 and above (prior to 5.1, Watcher integration requires putting together your own http attachment type with the correct headers, etc.).
Note: When configuring the watch, the URLs used in the reporting type are available from the Generation URL field under the Reporting sections for Discover and Dashboard in Kibana. The user specified requires reporting_user and kibana_user built-in roles as well as a custom role that provides sufficient privileges to query the indices backing the visualizations and saved searches in the report and CSV requested.
<pre class="prettyprint">{
 "trigger": {
 "schedule": {
 "interval": "1h"
 }
 },
 "actions": {
 "email_admin": {
 "email": {
 "to": "user@elastic.co",
 "subject": "My Report",
 "attachments": {
 "my_saved_search_export.csv": {
 "reporting": {
 "url": "http://localhost:5601/api/reporting/generate/csv?jobParams=(conflictedTypesFields:!(),fields:!(%27@timestamp%27,beat.hostname,beat.version,system.process.name,system.process.memory.rss.bytes,system.process.state),indexPatternId:%270c694c60-2975-11e8-a445-e50b6266a588%27,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!(),includes:!(%27@timestamp%27,beat.hostname,beat.version,system.process.name,system.process.memory.rss.bytes,system.process.state)),docvalue_fields:!(%27@timestamp%27),query:(bool:(filter:!(),must:!((match_all:()),(range:(%27@timestamp%27:(format:epoch_millis,gte:1365034274074,lte:1522800674075)))),must_not:!(),should:!())),script_fields:(),sort:!((%27@timestamp%27:(order:desc,unmapped_type:boolean))),stored_fields:!(%27@timestamp%27,beat.hostname,beat.version,system.process.name,system.process.memory.rss.bytes,system.process.state),version:!t),index:%27metricbeat*%27),title:%27New%20Saved%20Search%27,type:search)",
 "retries": 6,
 "interval": "15s",
 "auth": {
 "basic": {
 "username": "my_reporting_user",
 "password": "changeme"
 }
 }
 }
 },
 "my_dashboard.pdf": {
 "reporting": {
 "url": "http://localhost:5601/api/reporting/generate/printablePdf?jobParams=(browserTimezone:America%2FLos_Angeles,layout:(id:print),objectType:dashboard,queryString:%27_g%3D(refreshInterval:(display:Off,pause:!!f,value:0),time:(from:now-5y,mode:quick,to:now))%26_a%3D(description:!%27!%27,filters:!!(),fullScreenMode:!!f,options:(darkTheme:!!f,hidePanelTitles:!!f,useMargins:!!t),panels:!!((gridData:(h:3,i:!%271!%27,w:4,x:0,y:0),id:!%27149801f0-2a1b-11e8-a445-e50b6266a588!%27,panelIndex:!%271!%27,title:!%27My%2BPie%2BChart!%27,type:visualization,version:!%276.2.1!%27),(gridData:(h:3,i:!%273!%27,w:8,x:4,y:0),id:!%276383d890-3770-11e8-b3bb-7fbbe39bc802!%27,panelIndex:!%273!%27,title:!%27My%2BSaved%2BSearch!%27,type:search,version:!%276.2.1!%27)),query:(language:lucene,query:!%27!%27),timeRestore:!!t,title:!%27My%2BDashboard!%27,viewMode:edit)%27,savedObjectId:b9627b10-2a1c-11e8-a752-737beb0df489)",
 "retries": 6,
 "interval": "15s",
 "auth": {
 "basic": {
 "username": "my_reporting_user",
 "password": "changeme"
 }
 }
 }
 }
 }
 }
 }
 }
}
</pre><h3>You get a new layout mode!</h3>Prior to Kibana 6.1, visualizations were always formatted in a print friendly layout with two visualizations per A4 letter page. We added a new Preserve existing layout in PDF option in 6.1 that will produce a PDF report with the size and layout of visualizations preserved from the dashboard.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltac7b4fd30a3a50e5/5c306932c05f480270a74396/preserve_layout.png" data-sys-asset-uid="bltac7b4fd30a3a50e5" alt="preserve_layout.png" style="display: block; margin: auto;">
<h3>You get a custom logo!</h3>You can configure a custom logo to use in PDF reports (instead of the Elastic default) by setting <code>xpackReporting:customPdfLogo</code> under Management &gt; Advanced Settings in Kibana starting in version 6.2 and above. Avocado toast anyone?
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7a74f39ff1317a23/5c30692d25a401fc6fe4f539/avocado_footer.jpg" data-sys-asset-uid="blt7a74f39ff1317a23" alt="avocado_footer.jpg" style="display: block; margin: auto;">
<h3>You get a new reporting backend (in a future release)!</h3>Kibana currently uses <a href="http://phantomjs.org/">PhantomJS</a> as the default reporting backend. PhantomJS has been a great open source project, providing a headless browser for automating web page interaction. Unfortunately, the PhantomJS project is <a href="https://groups.google.com/forum/#!msg/phantomjs/9aI5d-LDuNE/5Z3SMZrqAQAJ">no longer being maintained</a> and development has been suspended. As a result, Elastic is actively working on making <a href="https://developers.google.com/web/updates/2017/04/headless-chrome">Chromium</a> (an open source project started by Google that forms the basis for Chrome) the default reporting backend in the near future. In versions 6.1 and above, you can actually opt-in to switch the default reporting browser type from PhantomJS to Chromium by setting <code>xpack.reporting.capture.browser.type: chromium</code> in <code>kibana.yml</code>.
User feedback from the field has been positive so far. For example, switching to Chromium has resolved various reporting issues such as truncated legends, chart misalignment, and missing axis labels (examples below):
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd2bf97bdf843d7ca/5c306961ca38a16366a3806e/floating_bars.png" data-sys-asset-uid="bltd2bf97bdf843d7ca" alt="floating_bars.png" style="display: block; margin: auto;">
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcee1e944f350ef88/5c306965e71ce40c6e4ada15/cut_off_legend.png" data-sys-asset-uid="bltcee1e944f350ef88" alt="cut_off_legend.png" style="display: block; margin: auto;">
<h2><a name="issues"></a>Troubleshooting Common Issues</h2><h3>How to resolve: Junk characters in visualization labels.</h3>PhantomJS has various font package dependencies. If you are running on Linux, you must install the <code>libfontconfig</code> and <code>libfreetype6</code> packages (Windows should already come with the necessary fonts). Without these dependent font libraries, you can see junk characters in report visualizations:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfc71823ee9109e99/5c30691ac05f480270a74390/boxes_not_letters.png" data-sys-asset-uid="bltfc71823ee9109e99" alt="boxes_not_letters.png" style="display: block; margin: auto;">
<h3>How to resolve: You must install freetype and ttf-font for Reporting to work.</h3>If you have switched the default reporting backend to Chromium, the font package dependencies are slightly different. In addition to installing <code>libfontconfig</code> and <code>libfreetype6</code> packages, chromium also requires additional font packages to be installed based on the TrueType standard for the specific OS/distro installed. For example, <a href="http://www.stixfonts.org/install.html">stix-fonts</a> will install otf files which are based on the TrueTyle standard. Without TrueType (or equivalent) fonts installed, you will receive the following error when generating a PDF report: You must install freetype and ttf-font for reporting to work.
<h3>How to resolve: Unable to use Chromium sandbox.</h3>This can happen if the kernel of the machine Kibana is running version &lt;= 3.8 (which does not support user namespaces, per <a href="https://chromium.googlesource.com/chromium/src/+/lkcr/docs/linux_sandboxing.md">Chromium documentation</a>). While you can disable sandbox via <code>xpack.reporting.capture.browser.chromium.disableSandbox: true</code> as a workaround, we recommend upgrading the kernel (unless you are running within a docker container). If Kibana is running on an older kernel with no user namespace support and with sandbox enabled, you will receive the following error when generating a PDF report: Unable to use Chromium sandbox.
<h3>How to resolve: Failed to decrypt report job data. Please re-generate this report.</h3>This error happens when the <code>xpack.reporting.encryptionKey</code> in <code>kibana.yml</code> is set to one value when the report is originally requested, but is another value by the time it ends up being processed. Here are the common causes:
<ul>
	<li><code>xpack.reporting.encryptionKey</code> was never set initially (so it gets randomly generated on startup) and Kibana was restarted</li>
	<li><code>xpack.reporting.encryptionKey</code> was set initially but manually changed and Kibana was restarted</li>
	<li>There are multiple instances of Kibana running Reporting all referencing the same Elasticsearch cluster, but one or more instances is/are using a different <code>xpack.reporting.encryptionKey</code></li>
</ul>To avoid this problem, always set <code>xpack.reporting.encryptionKey</code> to an explicit value that is consistently set across all Kibana instances (if connecting to the same Elasticsearch cluster).
<h3>How to handle long report generation time</h3>For reports with many complex visualizations or if the Kibana machine is under load, a reporting worker may take over 30s to generate a report and subsequently kill further report generation with the error: Max attempts reached (3). This timeout can be tuned by setting <code>xpack.reporting.queue.timeout</code> in the <code>kibana.yml</code>.
In earlier versions of reporting, there were also unrelated issues where empty images or a “loading …” indicator were captured as screenshots instead of the actual visualizations. A lot of these issues have been addressed in 6.2 and above.
<h3>How to enable debug logging
</h3>Debug logging for reporting can be enabled by adding the following lines to <code>kibana.yml</code>. This will produce additional log entries such as underlying API calls, workers, screenshot images created, and more.
<pre class="prettyprint">logging.verbose: true
logging.events.log: [ 'reporting' ]
</pre><h3>How to review the .reporting-* indices</h3>You can request details (eg. max attempts, priority, timeout, payload, etc..) of historical report generations by querying the system daily .reporting-* indices.
Note: This requires additional read privilege against .reporting-* indices for your user role (the built-in reporting_user role is not sufficient).
<pre class="prettyprint">GET .reporting-*/_search
{
 "_source": {
 "excludes": [
 "output.content",
 "payload.headers"
 ]
 },
 "sort": [
 {
 "created_at": {
 "order": "desc"
 }
 }
 ]
}
</pre><h3>How to delete older reports</h3>Currently, Kibana does not provide a way to delete older reports from the report listing. However, you can (carefully) delete them directly from the .reporting-* indices if required using a user with the superuser role or a user with delete privileges against the .reporting-* indices noted above. As a best practice, always <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-snapshots.html">backup</a> the indices before deleting any documents from them.
Hope you find this blog post helpful and happy reporting!

blog-kibana-banner.jpg

avocado_toast_thumb.jpg

'Cause We're Living in a (Reporting) World

This blog post talks about how you can use Reporting and Watcher integration on Elastic Cloud to schedule the delivery of PDF reports.

blog-reporting-fullbleed.jpg

blog-reporting-thumb.jpg

You get a report! You get a report!

This blog post talks about how you can use the Elastic Stack to search, discover and visualize data from Zendesk.

<html><head></head><body><a href="https://www.zendesk.com" target="_blank">Zendesk</a> is a popular lightweight helpdesk solution. In this blog post, I will demonstrate how to use the Elastic Stack to provide discovery and visualization capabilities on data that is extracted from Zendesk via a custom Logstash plugin.
<h2>Logstash: Gather What Matters (Not Just For Logs)</h2>Logstash is often used for log collection. However, it is highly extensible and has an intuitive plugin framework developers can use to build their own plugins to fetch data from other sources. To give a perspective on what it will take to build a custom Logstash plugin: I am a software support engineer (not a seasoned developer) with some experience writing internal tools and integrations. With no prior Ruby knowledge, I was able to follow the <a href="https://www.elastic.co/guide/en/logstash/current/_how_to_write_a_logstash_input_plugin.html" target="_blank">Logstash plugin development guide</a> and write a custom Logstash Zendesk plugin that fetches objects from Zendesk using the official <a href="https://github.com/zendesk/zendesk_api_client_rb" target="_blank">Zendesk Ruby API Client</a>. The majority of the time (~60%) was actually spent on learning the basics of Ruby and interaction with the Zendesk REST API and Zendesk Ruby API Client. The rest was writing code, testing and tweaking the implementation.
The custom/community <a href="https://github.com/ppf2/logstash-input-zendesk" target="_blank">Logstash Zendesk plugin</a> mentioned above supports fetching various Zendesk objects like organization, user, ticket, comment and topic. The following is an example input block (refer to the Github repository linked above for additional details). You can control the types of objects fetched, fetch mode (full or incremental based on last updated time) and whether to append comments to the ticket in chronological order, etc.
<pre class="prettyprint">input 
{ 
 zendesk 
 { 
 domain =&gt; &quot;company.zendesk.com&quot; 
 user =&gt; &quot;user@company.com&quot; 
 # password =&gt; &quot;your_password&quot; 
 api_token =&gt; &quot;your_api_token&quot; 
 organizations =&gt; true 
 users =&gt; true 
 tickets =&gt; true 
 tickets_last_updated_n_days_ago =&gt; 1 
 comments =&gt; true 
 append_comments_to_tickets =&gt; true 
 topics =&gt; true 
 sleep_between_runs =&gt; 60 
 } 
}
</pre>And the logging output of the plugin at --verbose level:
<img src="https://api.contentstack.io/v2/assets/575e4d129985d58976378086/download?uid=blt18cb181f1c460243?uid=blt18cb181f1c460243" style="max-width:100%;" data-sys-asset-uid="blt61b65ddcf0225215" alt="verbose level">
<h2 rel="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Kibana: Open Up Your Mind</h2>With the Zendesk data ingested into Elasticsearch via Logstash, you can use the popular <a href="https://www.elastic.co/guide/en/kibana/current/index.html" target="_blank">Kibana</a> tool to search, discover and visualize the data. For example, here&#x2019;s a date histogram aggregated by the severity levels of tickets over time. Imagine the other possibilities such as returning the top N customers mentioning keywords like ldap, hadoop, etc.. in their comments.
<img src="https://api.contentstack.io/v2/assets/575e4d123dc542cb38c09931/download?uid=blt787c1f4b44e9dac8?uid=blt787c1f4b44e9dac8" style="max-width:100%;" data-sys-asset-uid="blte3f898842bc55f64" alt="Kibana tool to search">
<img src="https://api.contentstack.io/v2/assets/575e4d12d8edd48f76937484/download?uid=blt78382eae9ca41eda?uid=blt78382eae9ca41eda" style="max-width:100%;" data-sys-asset-uid="bltc173642f780b1084" alt="Kibana dashboard">
<h2>Watcher: Help Others Be Mindful</h2>With the commercial plugin (<a href="https://www.elastic.co/downloads/watcher" target="_blank">Watcher trial</a>), you have endless opportunities to create watches not just to help streamline support operations (eg. tracking high severity and aging tickets, etc..). With the power of text search and aggregations, you can also monitor and alert on product quality trends. 
For example, the following watch&#x2019;s <a href="https://www.elastic.co/guide/en/watcher/current/input.html#input-search" target="_blank">search input action</a> identifies tickets with comments mentioning out of memory conditions in the past 30 days and aggregates the result by week showing the top 10 tickets sorted by version and date. This will produce a payload that can be <a href="https://www.elastic.co/guide/en/watcher/current/transform.html" target="_blank">transformed</a> to your desire and then sent to an output action. To whom do you send this to? Imagine setting up an output action for the payload to alert the product management team using their <a href="https://www.elastic.co/guide/en/watcher/current/actions.html#actions-email" target="_blank">email alias</a> or <a href="https://www.elastic.co/guide/en/watcher/current/actions.html#actions-hipchat" target="_blank">HipChat</a>/<a href="https://www.elastic.co/guide/en/watcher/current/actions.html#actions-slack" target="_blank">Slack</a> room so they can determine if there are areas in the product that can be improved (eg. circuit breakers). 
<img src="https://api.contentstack.io/v2/assets/575e4d123dc542cb38c0993f/download?uid=blt8bdc7df7016da19e?uid=blt8bdc7df7016da19e" style="max-width:100%;" data-sys-asset-uid="blt13ea018e11064b25" alt="circuit breakers">
<h2>Shield: Block Out Disturbances</h2>You don&#x2019;t want your customer data to be wide open. Use the commercial plugin (<a href="https://www.elastic.co/downloads/shield" target="_blank">Shield trial</a>) to require <a href="https://www.elastic.co/guide/en/shield/current/setting-up-authentication.html" target="_blank">authentication</a> and access control, and set up <a href="https://www.elastic.co/guide/en/shield/current/securing-communications.html" target="_blank">encryption</a> to further secure your data. Classified customers? No problem, Shield provides more granular role based access control via <a href="https://www.elastic.co/guide/en/shield/current/setting-up-field-and-document-level-security.html" target="_blank">document and/or field level security</a>.
<h2>Found: Find Peace Of Mind</h2>Deploying the Elastic Stack on <a href="https://www.elastic.co/found/signup" target="_blank">Found</a> is quick and simple. It literally takes just minutes to spin up a Found cluster provisioned with the latest and greatest Elastic releases. Since it is important to prevent unauthorized access to the data set, you can choose <a href="https://www.elastic.co/found" target="_blank">Found Premium</a> which includes the commercial plugins mentioned above (Shield and Watcher). Once the Found cluster has started, you can simply configure the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html" target="_blank">Logstash Elasticsearch output</a> in your Logstash pipeline to reference the Found cluster&#x2019;s host(s), Shield user and password, and start indexing! The data will begin to appear in the Kibana app.
<img src="https://api.contentstack.io/v2/assets/575e4d129985d58976378094/download?uid=blt82d704528733afe0?uid=blt82d704528733afe0" style="max-width:100%;" data-sys-asset-uid="blt7ea02a33f538cb20" alt="Kibana app">
The best part is that Found is a managed service by the creators of the Elastic Stack so you will not have to worry about the uptime of the cluster or spend hours explaining to your infrastructure team what you have deployed or waiting for them to perform operational tasks. It also takes automatic/periodic backups of your indices. 
<img src="https://api.contentstack.io/v2/assets/575e4d1243e9adc538715106/download?uid=bltc956d04b4eaa16e3?uid=bltc956d04b4eaa16e3" style="max-width:100%;" data-sys-asset-uid="blt1246cc29722e65ae" alt="periodic backups">
This means you will have more time to build impressive dashboards, work on your regular job, or walk the dog :)As you can see, the Elastic Stack is applicable to many different use cases, not limited to logging. Its flexibility is in its name. With Found, you can also get up and running quickly!</body></html>

stacked-stones-water.jpg

stacked-stones-water-thumb.jpg

Practicing The Art of Zen(desk) With The Elastic Stack

Can I run an Elasticsearch cluster across multiple data centers?

<html><head></head><body>We are frequently asked whether it is advisable to distribute an Elasticsearch cluster across multiple data centers (DCs). The short answer is &quot;no&quot; (for now), but there are some alternate options available described below. This blog post is intended to help you understand why this is the case, and what other options are available to you. <img src="https://api.contentstack.io/v2/assets/575e4cdc9e7a83165490ce37/download?uid=blt7e3ce97d21934a40?uid=blt7e3ce97d21934a40" data-sys-asset-uid="bltccb0142e49aa5f2a" alt="multi-DC.png" width="450" height="255" style="width: 450px; height: 255px;"><h2 dir="ltr"> But Why? </h2>The architecture and design decisions that we make in Elasticsearch are based on certain assumptions, including the assumption that nodes are located on a local network. This is the use case that we optimize and extensively test for, because this is the environment that the vast majority of our users operate in. Network disruptions are much more common across WAN links, between geographical distributed DCs. Even if there is a dedicated link between DCs. &#xA0;Elasticsearch is built to be resilient to networking disconnects, but that resiliency is intended to handle the exception, not the norm. Running a single Elasticsearch cluster that spans multiple DCs is not a scenario we test for and there are a number of additional reasons why it is not a recommended or supported practice we will go into below.&#xA0; (Note: &#xA0;On GCE and AWS, running a cluster across zones within a single region is supported) <h2 dir="ltr">Expect the Unexpected </h2>High Latency Latency is a problem in distributed systems. &#xA0;&#xA0;High latency slows indexing because the indexing request is <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/distrib-write.html">processed on the primary shard first and then sent to all the replicas</a> for indexing, and all cluster-wide communications (eg. cluster state updates) in Elasticsearch. Limited or Unreliable Connectivity If connectivity between nodes in a cluster is momentarily lost, it&#x2019;s likely that remote shards will be out of date and any single update processed while in disconnected state will invalidate all content held on isolated replicas. This means that Elasticsearch requires the copying of these out of date shards to sync up replicas from their primaries to ensure consistency of data and search responses. Sending full shards for multiple indices may overwhelm a WAN based connection or cause considerable slowdown, leaving your cluster in a degraded state for an extended period of time. Data Availability Assuming the correct setting of <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html#_minimum_master_nodes">discovery.zen.minimum_master_nodes</a>, in the event of a network disconnect between two or more DCs, only the DC with the elected master node will remain active. This can cause many issues for applications in the different DCs which may be attempting to index new data, as the nodes not part of the active cluster will reject any attempted writes. This also provides a challenge with cluster sizing. When the link between the two DCs is broken, the active half of the cluster will need to bear the full load of indexing and queries for all requests. When the link is restored, these nodes will also be pushing data and documents across the network while still handling the full indexing and request load. This necessitates larger or more powerful clusters to ensure enough CPU and IOPS to maintain acceptable performance during such events. <h2 dir="ltr">What Are The Options ? </h2>Here are the&#xA0;common scenarios on how this may look to give you some ideas. Real-time Availability Across Geographic RegionsHere you would have your application code write to a replicated queuing system (e.g. Kafka, Redis, RabbitMQ) and have a process (e.g. <a href="https://www.elastic.co/guide/en/logstash/current/index.html">Logstash</a>) in each DC reading from the relevant queue and indexing documents into the local Elasticsearch cluster.This way if network connectivity is lost between the DCs, when it is restored, the indexing will continue where it left off. Simple Disaster Recovery (DR) with No Hot Replication Requirements <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#modules-snapshots">Snapshots</a> can be used to backup indices at regular intervals (eg. to S3) and restore to the passive DC for disaster recovery. Tools such as <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/snapshot.html">Elasticsearch Curator</a> make automation of the Snapshot process very simple. Restoration can either be done as soon as the Snapshot has completed or at scheduled times such as hourly or even just once daily.The Snapshot and Restore only copies the segments files that do not already exist in the snapshot repository, so except for the initial snapshot, your backups are incremental which reduces the amount of space needed to store them.Even if you aren&#x2019;t using Snapshot and Restore to replicate data between DCs, we definitely recommend it to make sure you have a backup! Dataset Local to Each DCFor cases where the data is local to each DC (eg. 1 dataset in Hong Kong, another in London) and there is a need to search across all of them, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-cross-cluster-search.html#modules-cross-cluster-search" target="_blank">cross cluster search</a>&#xA0;can be used to give a view of both datasets. This allows you to query both clusters as if they were one big cluster, reducing complexity in having to merge datasets. &#xA0;Note that master write operations like create index should be performed against the individual clusters. Cross Cluster Replication Elastic <a href="https://www.elastic.co/subscriptions">subscription</a> customers can use the new <a href="https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-ccr.html" target="_blank">cross cluster replication</a> feature to easily replicate indices between a primary and a secondary (backup) cluster, or serve reads locally for geo-proximity. <h2 dir="ltr">Final Words</h2> As you can see, this is an area that we think a lot about, and we continue to keep these in mind as we design and develop the multi-cluster replication functionality that is on the Elasticsearch roadmap.</body></html>

AUTHOR

Articles by Pius Fung

Monitoring Elastic Enterprise Search performance using Elastic APM

'Cause We're Living in a (Reporting) World

You get a report! You get a report!

Practicing The Art of Zen(desk) With The Elastic Stack

Clustering Across Multiple Data Centers

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Pius Fung

Monitoring Elastic Enterprise Search performance using Elastic APM

'Cause We're Living in a (Reporting) World

You get a report! You get a report!

Practicing The Art of Zen(desk) With The Elastic Stack

Clustering Across Multiple Data Centers

Sign up for Elastic Cloud free trial

We're hiring