This article outlines several approaches to migrate to Elastic's very own Elastic Cloud.

<p>Whether you’re trying to reduce operational costs, maintain uptime or generally simplify things, consider <a href="https://www.elastic.co/cloud">Elastic Cloud</a>, Elastic’s very own hosted Elasticsearch and Kibana offering. Elastic Cloud offers ease-of-use and provides the latest Elasticsearch and Kibana releases, version 5.1 at the time this was written, with the newest features, enhancements and performance improvements. Also included are X-Pack features, such as security, monitoring, alerting & notification, Graph, and reporting. Elastic Cloud is backed by Elastic’s excellent support team and by the engineers who develop Elasticsearch and Kibana.<br></p><p>Interested in trying out Elastic Cloud? Let us show you how you can migrate your data to our hosted offering, regardless of whether you currently self-host Elasticsearch on your own metal, run Elasticsearch in some other cloud offering, such as EC2, or use one of the many Elasticsearch hosting services that exist, such as the Amazon Elasticsearch Service.
</p><h2>Migration Approaches</h2><p>If you have existing Elasticsearch indexes and are considering migrating to Elastic Cloud, there are a few approaches to consider. Regardless of the approach, keep in mind that you first have to configure your Elastic Cloud target cluster. Creating a new cluster is very easy, but cluster settings like scripts and index settings like custom analyzers and mappings aren’t automatically copied over and must be configured prior to migrating index data.
</p><h3>Feed From The Source</h3><p>If you have a copy of the original data some place other than your Elasticsearch cluster, then it might be simplest to load from there. Create an Elastic Cloud cluster, configure it, and feed away. Ingesting into an Elastic Cloud cluster is no different than feeding a cluster running on your own servers. You can use <a href="https://www.elastic.co/products/logstash">Logstash</a>, <a href="https://www.elastic.co/products/beats">Beats</a>, the <a href="https://www.elastic.co/guide/en/elasticsearch/client/index.html">Elasticsearch Clients</a> or any other means at your disposal.
</p><p>If the original source isn’t available, is cumbersome to access, has changed, or is no longer up to date, then two other migration approaches are available: reindexing from remote or snapshotting and restoring.
</p><h3>Reindex From A Remote Cluster</h3><p>Elasticsearch 5.x provides the ability to index from a remote cluster with the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/docs-reindex.html#reindex-from-remote">reindex API</a>. With reindexing, it’s possible to spin up a cluster in Elastic Cloud and index data from an existing remote cluster. First create the destination index on the cluster in Elastic Cloud and then use reindex with the `remote` option to pull data from your old cluster and index it into the current cluster, taking advantage of the latest <a href="https://www.elastic.co/blog/elasticsearch-5-0-0-released#data-structures">data structures</a>, <a href="https://www.elastic.co/blog/elasticsearch-5-0-0-released#resiliency">resiliency</a> and <a href="https://www.elastic.co/blog/elasticsearch-5-0-0-released#indexing">indexing performance</a> Elasticsearch 5.1 improvements.
</p><p>Reindexing effectively rebuilds the index from scratch so it is more expensive to run than _snapshot, which simply copies complete indexes. The tradeoff is that restored indexes can't take advantage of those nice data structures because they are not rebuilt.
</p><h3>Restore Snapshots From S3</h3><p>It’s also possible to <a href="https://www.elastic.co/guide/en/cloud/current/snapshot-restore.html">restore Elasticsearch snapshots sitting in S3 buckets</a>. With this approach, snapshot to an S3 bucket from an existing Elasticsearch cluster, spin up an Elasticsearch cluster in Elastic Cloud, and then restore the snapshot from S3 into Elastic Cloud.
</p><p>Bear in mind that you’re not limited to restoring snapshots from S3 buckets. Snapshots that exist on any addressable external storage can be restored into Elastic Cloud.
</p><h2>How to Reindex from Remote</h2><h3>Reindex from Elasticsearch on EC2 into Elastic Cloud</h3><p>Log into <a href="https://www.elastic.co/cloud">Elastic Cloud</a> and create a new cluster, minimally the same size as your Elasticsearch on EC2 cluster or larger if you anticipate growth. Be sure to enable Kibana so you have access to the console.
</p><p>From your Elastic Cloud cluster, you can issue the reindex command to index data stored remotely in Elasticsearch on EC2. The reindex command requires access to port 9200 or 9243, so you should <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html">update the Security Group</a> associated with the EC2 Elasticsearch cluster and add an inbound custom TCP entry for port 9200 or 9243, allowing access to the Elastic Cloud clusters.
</p><p>Issue the reindex command:
</p><pre class="prettyprint">POST reindex
{
  "source": {
    "remote": {
      "host": "http://[ec2 public hostname]:9200"
    },
    "index": "bank",
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "bank"
  }
</pre><p>Verify that your documents were indexed:
</p><pre class="prettyprint">GET /bank/accounts/_search?pretty
</pre><h3>Reindex from the Amazon Elasticsearch Service into Elastic Cloud</h3><p>Presently, the most recent Elasticsearch version available on the Amazon Elasticsearch Service is 2.3, a major version behind the current Elasticsearch release. If you would like to take advantage of the most recent Elasticsearch 5.1 features, you can reindex directly into Elastic Cloud from the Amazon Elasticsearch Service.
</p><p>The reindex command is the same used in the Elasticsearch on EC2 example, with one minor change. The Amazon Elasticsearch Service proxies requests to port 9200 or 9243, so it’s necessary to specify port 80 (HTTP) or port 443 (HTTPS) with the host. Not providing the port will result in an error.
</p><h3>HTTPS Example</h3><pre class="prettyprint">POST reindex
{
  "source": {
    "remote": {
      "host": "https://[aws es public hostname]:443"
    },
    "index": "bank",
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "bank"
  }
}
</pre><h2>How to Restore Snapshots to An Elastic Cloud Cluster</h2><p>In this example, we’ll show you how to snapshot an index from EC2 into an S3 bucket and then restore the snapshot from the S3 bucket into an Elastic Cloud cluster. One thing to note is that S3 buckets shared between EC2 and Elastic Cloud clusters need to reside in the same region.
</p><h3>Step I - Snapshot from Elasticsearch on EC2 into S3</h3><p>First you’ll need to create an S3 bucket using AWS. In this example, we’ve created an S3 bucket using the name for the region "US Standard". “US Standard” is synonymous with “US-East”.
</p><p>You’ll need to edit the bucket policy and should use these <a href="https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-repository.html#repository-s3-permissions">recommended S3 permissions</a>:
</p><pre class="prettyprint">{
    "Id": "Policy1477318346906",
    "Version": "2012-10-17",
    "Statement": [
    {
        "Sid": "Stmt1477318310202",
        "Action": "s3:*",
        "Effect": "Allow",
        "Resource": "arn:aws:s3:::[bucket name]/*",
        "Principal": "*"
    }
    ]
}
</pre><p>Next, we’ll need to create a snapshot repository within the Elasticsearch cluster running on EC2:
</p><pre class="prettyprint">curl -XPUT 'localhost:9200/_snapshot/[repository name]?verify=false&pretty' -d'
{
  "type": "s3",
  "settings": {
    "access_key": "[access key]",
    "secret_key": "[secret key]",
    "bucket": "[bucket name]"
  }
}'
</pre><p>Now, let’s go ahead and snapshot the EC2 index into the repository:
</p><pre class="prettyprint">curl -XPUT  'localhost:9200/_snapshot/[repository name]/snapshot_1?pretty' -d'
{
    "indices": "*",
    "ignore_unavailable": true,
    "include_global_state": true
}'
</pre><p>Now, verify the index snapshot exists in S3 by going into the AWS S3 console and viewing the contents of the bucket.
</p><h3>Step II - Restore Snapshot from S3 into Elastic Cloud</h3><p>First, we’ll need to create the Snapshot Repository for the Elasticsearch cluster running in Elastic Cloud:
</p><pre class="prettyprint">curl -u "elastic:ViOpqfQUW2KeU5n5SF1qzW7g" -XPUT '[elastic cloud instance host]:9200/_snapshot/[repository name]?verify=false&pretty' -d'
{
  "type": "s3",
  "settings": {
    "access_key": "[access key]",
    "secret_key": "[secret key]",
    "bucket": "[bucket name]"
  }
}'
</pre><p>Next, we’ll restore the snapshot on S3 into Elastic Cloud. If an index already exists with the name of the index being restored into, delete the index before restoring the snapshot.
</p><pre class="prettyprint">curl -u "elastic:[user id]" -XDELETE '[elastic cloud instance host]:9200/bank’
</pre><p>Now, restore the snapshot:
</p><pre class="prettyprint">curl -u "elastic:[user id]" -POST '[elastic cloud instance host]:9200/_snapshot/[repository name]/snapshot_1/_restore?pretty' -d'
{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": true
}'
</pre><p>Finally, verify the index has been restored into Elastic Cloud by running a query:
</p><pre class="prettyprint">curl -u "elastic:[user id]" -GET '[elastic cloud instance host]:9200/bank/_search?pretty'
</pre><p>While it’s possible to restore a snapshot from 2.4.1 into 5.1, if there are any index features that changed or were breaking changes, the restore will fail. Also, if they have any indices created in 1.x, those can’t be restored into 5.1.
</p><p>The safest approach is to restore into the same version, then upgrade to 5.1.
</p><h2>Summary</h2><p>Ready to give it a try? A <a href="https://www.elastic.co/elasticsearch/service">free 14 day trial</a> is available on us. No credit card required.
</p><p>Ready to make the switch but not sure about pricing? Pricing is a snap with the <a href="https://www.elastic.co/cloud/as-a-service/pricing">pricing calculator</a>. Select cluster size, region and high availability preferences and the price is calculated instantly.</p>

elastic-cloud-tutorial-full-bleed.jpg

elastic-cloud-thumb.jpg

Migrating to Elastic Cloud

<p>Ever-increasing quantities and varieties of data are being created and captured. Application, system and user behavior logs, network packets, sensor metrics... you name it. The availability of these data drives a diverse and vast set of use cases ranging from intrusion detection to user behavior analysis, network monitoring, machine monitoring, and remote sensor monitoring, just to name a few.</p><p><a href="/products/beats">Beats</a> are open source, purpose-built, lightweight, and efficient agents that acquire and feed data natively to Elasticsearch. Optionally, Beats can feed data to Logstash for further refinement before being forwarded to Elasticsearch.</p><p>As part of the Elastic Stack, Beats is a <a href="https://en.wikipedia.org/wiki/Telemetry" target="_blank">telemetry</a> architecture, enabling the capture and transmission of measurements and other data from remote sources to Elasticsearch for analysis, aggregation, and search. Anywhere there is change, such as production and consumption, growth and demise, acceleration and deceleration, data has significance. GPS-driven tractors capture soil depths, soil fertility, and ground temperatures, all of which influence agricultural strategy. Ocean buoys record wave amplitudes that may help to warn of impending catastrophes or inform of welcome beach conditions. Automobiles capture thousands of metrics ranging from tire pressure to average speed to fuel consumption that facilitate an understanding of a vehicle's abilities and failures, as well as the owner's driving behaviors. Network sensors capture machine communication messages that help with understanding usage patterns that drive security and infrastructure policies.</p><p>In the previous article <a href="/blog/running-elasticsearch-on-aws">Running Elasticsearch on AWS</a> we configured a three-node Elasticsearch cluster on AWS EC2. It was fairly straightforward and didn't take much time or effort. We will expand on the <a href="/blog/running-elasticsearch-on-aws">Running Elasticsearch on AWS</a> example and configure Beats to feed the Elasticsearch cluster running on AWS EC2 instances. This will demonstrate how simple it is to configure a centralized machine resource monitoring solution.</p><p>First, we'll provide an overview of Beats, review available Beats and the environments in which they run - Part I. Then we'll do a step-by-step example installing, configuring, and verifying Beats output - Part II.</p><h2>Part I - Beats Overview</h2><p><a href="/products/beats">Beats</a> are purpose-built lightweight data shippers, or agents, that run on remote machines and feed Elasticsearch instances. Beats make it easy to get data into Elasticsearch. Beats are available on a number of operating systems such as Debian, Redhat, Linux and Mac.</p><p>There are several available <a href="/products/beats">Beats</a> including Filebeat, Metricbeat, Packetbeat, Winlogbeat and Topbeat. Each Beat has a specific purpose or multiple purposes that are logically related, allowing each Beat to focus on its specific task and do it well.</p><p><a href="/products/beats/filebeat">Filebeat</a> tails logs and can ship data to Logstash for further refinement, or directly to Elasticsearch for analysis and search. Filebeat can be installed on any machine that has applications that generate log data, such as a database or application server.</p><p><a href="/downloads/beats/metricbeat">Metricbeat</a> (Alpha) captures operating system metrics as services such as Apache web server and Redis. Metricbeat does everything Topbeat does and much more in that it captures operating system metrics such as per-process CPU, memory, and storage use as well as common application messages, such as web servers. Metricbeat will replace Topbeat, so you might consider using Metricbeat for development.</p><p><a href="/products/beats/packetbeat">Packetbeat</a> captures web, database, and other network protocols, enabling Kibana real-time analytics. Packetbeat is extensible, enabling the addition of new protocols, metrics and analytics.</p><p><a href="/downloads/beats/winlogbeat">Winlogbeat</a> captures Windows event log system, application and security data, enabling monitoring of Windows machines.</p><p><a href="/products/beats/topbeat">Topbeat</a> captures per-process memory, CPU and disk usage statistics. Topbeat can be installed across a set of machines and indexed into a central Elasticsearch cluster, enabling centralized machine resource monitoring. Topbeat is being replaced by Metricbeat, currently in Alpha, and the change is discussed in this <a href="/blog/beats-5-0-0-alpha3-released">article</a>.</p><p>If you have metric data not addressed by available Beats, there's also <a href="https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html" target="_self">Libbeat</a>, a framework for developing new Beats.</p><p>The <a href="https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html">Beats Community</a> is growing and developers are contributing everything from pingbeat, for capturing ping roundtrips, to redisbeat, for Redis monitoring, and to hsbeat, for capturing Java HotSpot VM performance counters.</p><h2>Part II - Configuring Beats</h2><p>In this example we will install <a href="https://www.elastic.co/products/beats/topbeat">Topbeat</a> and <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-overview.html">Packetbeat</a> on a development machine, in this case a Macbook Pro, and feed an Elasticsearch cluster running on AWS. Beats can be installed on any machine running a supported operating system that has network connectivity with the Elasticsearch cluster.</p><h3>Configure Topbeat</h3><p><a href="/downloads/beats/topbeat">Download Topbeat</a> to a directory on the machine that will be running the Beats.</p><p>Download and extract the Topbeat archive using your preferred method, for example:</p><pre class="prettyprint">curl -L -O &lt;a href="https://download.elastic.co/beats/topbeat/topbeat-1.2.3-darwin.tgz"&gt;https://download.elastic.co/beats/topbeat/topbeat-...&lt;/a&gt;</pre><pre class="prettyprint">tar xzvf topbeat-1.2.3-darwin.tgz<br /></pre><p><img src="https://api.contentstack.io/v2/assets/57853a1ffc8134614f684651/download?uid=blt8fbd3baf06626f6d" data-sys-asset-uid="blt8fbd3baf06626f6d" alt="get_topbeat.png"/></p><p>This will create a folder named topbeat-1.2.3-darwin that contains topbeat, topbeat.yml and topbeat.template.yml.</p><p>Locate and open topbeat.yml in your preferred editor. There is a <em>hosts</em> property that tells Topbeat where Elasticsearch is located. Locate the hosts parameter and update it to refer to the AWS public hostname, for example:</p><p><img src="https://api.contentstack.io/v2/assets/57853a36b4a7391d0c76deed/download?uid=blt1c456161684b4a17" data-sys-asset-uid="blt1c456161684b4a17" alt="topbeat_hosts.png"/></p><p>Next, find the <em>template</em> property, which defines the index template that defines the field mappings required by Topbeat. Topbeat will load the index template file topbeat.template.json to Elasticsearch. Uncomment the template property along with the name and path properties that immediately follow.</p><p><img src="https://api.contentstack.io/v2/assets/57853a41fc8134614f684670/download?uid=bltedc473e2c49ece65" data-sys-asset-uid="bltedc473e2c49ece65" alt="topbeat_template.png"/></p><p>Then, run Topbeat with the command:</p><pre class="prettyprint">sudo ./topbeat -e -c topbeat.yml<br /></pre><p>To verify documents are being indexed, we'll use curl to make a search request:</p><pre class="prettyprint">curl -XGET 'http://ec2-50-17-114-78.compute-1.amazonaws.com:9200/topbeat-2016.07.07/_search?pretty'<br /></pre><p><img src="https://api.contentstack.io/v2/assets/57853a3bb4a7391d0c76def4/download?uid=bltaf4915db04f8e94d" data-sys-asset-uid="bltaf4915db04f8e94d" alt="topbeat_request.png"/></p><p>The curl request has several components. The first is the AWS EC2 public hostname "ec2-50-17-114-78.compute-1.amazonaws.com". This should be changed to your EC2 instance's public hostname. The second component "topbeat-2016.07.06" is the index. Lastly, is the search request "_search" with an optional "pretty" argument to format the JSON response.</p><p>If you want to dive deeper into Topbeat, a comprehensive <a href="https://www.elastic.co/guide/en/beats/topbeat/current/_overview.html">Topbeat Reference</a> is available and it covers topics such as <a href="https://www.elastic.co/guide/en/beats/topbeat/current/configuring-howto-topbeat.html">configuration</a> and <a href="https://www.elastic.co/guide/en/beats/topbeat/current/securing-topbeat.html">security</a>.</p><h3>Configure Packetbeat</h3><p><a href="/downloads/beats/packetbeat">Download Packetbeat</a> to a directory on the machine that will be running the Beats. Download and extract the Packetbeat archive using your preferred method, for example:</p><pre class="prettyprint">curl -L -O&lt;a href="https://download.elastic.co/beats/packetbeat/packetbeat-1.2.3-darwin.tgz"&gt;https://download.elastic.co/beats/packetbeat/packe...&lt;/a&gt;</pre><pre class="prettyprint">tar xzvf packetbeat-1.2.3-darwin.tgz<br /></pre><p><img src="https://api.contentstack.io/v2/assets/57853a18fc8134614f68464b/download?uid=blt4cd347d705e292c7" data-sys-asset-uid="blt4cd347d705e292c7" alt="get_packetbeat.png"/></p><p>This will create a folder named packetbeat-1.2.3-darwin that contains three files: packetbeat, packetbeat.yml, and packetbeat.template.yml.</p><p>Locate and open packetbeat.yml in your preferred editor. There is a <em>hosts</em> property that tells Packetbeat where Elasticsearch is located. Locate the hosts parameter and update it to refer to the AWS public hostname, for example:</p><p><img src="https://api.contentstack.io/v2/assets/57853a26fc8134614f684657/download?uid=blt738187754a29771b" data-sys-asset-uid="blt738187754a29771b" alt="packetbeat_hosts.png"/></p><p>Next, find the <em>template</em> property, which defines the index template that defines the field mappings required by Topbeat. Topbeat will load the index template file topbeat.template.json to Elasticsearch. Uncomment the template property along with the name and path properties that immediately follow.</p><p><img src="https://api.contentstack.io/v2/assets/57853a2cfc8134614f68465e/download?uid=bltbabfd10ee4a17877" data-sys-asset-uid="bltbabfd10ee4a17877" alt="packetbeat_template.png"/></p><pre class="prettyprint">sudo ./packetbeat -e -c packetbeat.yml<br /></pre><p>To verify documents are being indexed, we'll use curl to make a search request:</p><pre class="prettyprint">curl -XGET 'http://ec2-50-17-114-78.compute-1.amazonaws.com:9200/packetbeat-2016.07.07/_search?pretty'<br /></pre><p><img src="https://api.contentstack.io/v2/assets/57855336b4a7391d0c76e703/download?uid=blt6665d171b20e6547" data-sys-asset-uid="blt6665d171b20e6547" alt="packetbeat_request.png"/></p><p>If you want to dive deeper into Packetbeat, a comprehensive <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-overview.html">Packetbeat Reference</a> is available and it covers topics such as <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/configuring-howto-packetbeat.html">configuration</a> and <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/securing-packetbeat.html">security</a>.</p><h2>Part III - Summary</h2><p>Installing, configuring and running Beats is very easy. Beats address a broad set of use cases. They eliminate the challenge of acquiring data and feeding Elasticsearch.&nbsp;</p><p>Beats can feed any network accessible Elasticsearch cluster, whether it resides on Amazon AWS EC2, Microsoft Azure, dedicated iron and even <a href="https://www.elastic.co/cloud">Elastic Cloud</a>.</p><p>In the next installment of the AWS blog series, we'll cover installing Kibana on AWS EC2 and visualize the Beats indexed in this example.</p>

aws-beats-elasticsearch.png

aws-beats-elasticsearch-720x420.png

Using Beats with Amazon AWS

<p><a href="https://www.elastic.co/blog/get-hosted-elasticsearch-on-your-aws-bill-with-elastic-cloud">March 30, 2017 Update</a>: Elastic Cloud (<a href="https://aws.amazon.com/marketplace/pp/B01N6YCISK?ref=_ptnr_elastic_fb_blog_esawsmp">hosted Elasticsearch) on AWS</a> can now be added directly&nbsp;your AWS bill through the <a href="https://aws.amazon.com/marketplace/pp/B01N6YCISK?ref=_ptnr_elastic_fb_blog_esawsmp">marketplace</a>. If you are looking for hosted &amp; managed Elasticsearch, you can <a href="https://www.elastic.co/cloud/as-a-service/signup">try Elastic Cloud</a> for 14-days at no cost. For deploying and managing yourself on AWS EC2, this is the right article for you:</p><h2>Part I - Provisioning EC2 Instances</h2><p><em>Want to learn more about the differences between the Amazon Elasticsearch Service and our official Elasticsearch Service? Visit our </em><a href="/aws-elasticsearch-service#full-comparison"><em>AWS Elasticsearch</em></a><em> comparison page.</em></p><p>We often talk to customers running Elasticsearch clusters on Amazon Web Services (AWS). AWS is a convenient way to provision and scale machine resources in response to changing business requirements. Elasticsearch takes advantage of EC2's on-demand machine architecture enabling the addition and removal of EC2 instances and corresponding Elasticsearch nodes as capacity and performance requirements change.</p><p>In this article we will show you how to deploy <a href="/blog/elasticsearch-2-3-3-released">Elasticsearch 2.3.3</a> on Amazon EC2. In this example we will configure a three node Elasticsearch cluster.</p><h3>Step 1: Choose an Amazon Machine Image (AMI)</h3><p>Elasticsearch runs on various operating systems such as CentOS, Redhat, Ubuntu, and Amazon Linux. We suggest using the latest Amazon&nbsp;Linux AMI — "Amazon Linux AMI 2016.03.0 (HVM), SSD Volume Type".</p><p><img src="https://api.contentstack.io/v2/assets/576b089c1f82392051463860/download?uid=blt9963de9b246441e3" data-sys-asset-uid="blt9963de9b246441e3" alt="Choose an Amazon Machine Image (AMI).png" data-image="tpzk8so0uj9z"/></p><h3>Step 2: Choose an Instance Type</h3><p>A reasonable starting instance type is m3.2xlarge which provides 8 vCPUs, 30 GiB of memory, 2 x 80 GB SSD drives and comes with High Network Performance. Solid State Drives are preferred as indexing is IO intensive and High Network Performance is essential for cluster performance and reliability.</p><p>M3.2xlarge is a baseline recommendation. To determine whether it is an appropriate choice, you should <a href="/blog/announcing-rally-benchmarking-for-elasticsearch">benchmark your solution</a> to determine whether it meets performance and scaling requirements.</p><p>Click the "Next: Configure Instance Details" button.</p><p><img src="https://api.contentstack.io/v2/assets/576b088e1f82392051463854/download?uid=blt15d7ca9b0dac952f" data-sys-asset-uid="blt15d7ca9b0dac952f" alt="Choose an Instance Type.png" data-image="30nvr4jqsbaw"/></p><h3>Step 3: Configure Instance Details</h3><p>Each Elasticsearch node will run on its own dedicated EC2 instance, so set the number of instances to 3.</p><p>Note that any AWS accounts that have been created after December 4, 2013 only support EC2-VPC, so the "Network" option for picking "Launch into EC2-Classic" won't be available for those users and should not be enabled anyway.</p><p>Selecting "Enable termination protection" is a good idea as it prevents accidental deletion of nodes and their data.</p><p>Leave the default values for remaining fields and click the "Next: Add Storage" button.</p><p><img src="https://api.contentstack.io/v2/assets/576b09691f8239205146387f/download?uid=blt670618f63168180d" data-sys-asset-uid="blt670618f63168180d" alt="Configure Instance Details.png" data-image="k6avd8x47tes"/></p><h3>Step 4: Add Storage</h3><p>Let's leave the storage Size at 8 GiB. If you happen to know your index storage requirements at this time, you can adjust the storage now. Leave the Volume Type set to General Purpose SSD.</p><p>Click the "Next: Tag Instance" button.</p><p><img src="https://api.contentstack.io/v2/assets/576b089685453e6f4e3b4cb4/download?uid=blt661a2c609d1404bd" data-sys-asset-uid="blt661a2c609d1404bd" alt="Add Storage.png" data-image="cgztngs2si4j"/></p><h3>Step 5: Tag Instance</h3><p>In this field, provide a key and value pair, for example "name" and "esonaws", to make it easy to recall the ec2 instances.</p><p>Click the "Next: Configure Security Group" button.</p><p><img src="https://api.contentstack.io/v2/assets/576b083b173b7f5d5060324c/download?uid=blt09612d14527f408b" data-sys-asset-uid="blt09612d14527f408b" alt="Step 5 Tag Instance.png" data-image="8wv78nupahrq"/></p><h3>Step 6: Configure Security Group</h3><p>This configuration panel allows you to configure a set of firewall rules for accessing your instance. By default, Elasticsearch exposes TCP port 9200 for REST API access and TCP port 9300 for internal cluster communication. Consider adding rules to allow connecting to TCP port 9200 from desired subnets, typically private subnets, and TCP port 9300 from the subnets where Elasticsearch nodes live. If you plan to change the default port settings in elasticsearch.yml, configure rules for those ports rather than TCP ports 9200 and 9300.</p><p>Also, add a rule to allow SSH connections on port 22, so you can connect to the instance in the later steps.</p><p>Click The "Review and Launch" button.</p><p><img src="https://api.contentstack.io/v2/assets/576b085685453e6f4e3b4ca3/download?uid=bltb5d9899fe69a7840" data-sys-asset-uid="bltb5d9899fe69a7840" alt="Configure Security Group.png" data-image="ndo0u0se0rzm"/></p><h3>Step 7: Review Instance Launch</h3><p>Note any warnings and review the Instance Launch settings and click the "Launch" button when ready.</p><p><img src="https://api.contentstack.io/v2/assets/576b08481f8239205146383c/download?uid=bltc258259f2721b723" data-sys-asset-uid="bltc258259f2721b723" alt="Review Instance Launch.png" data-image="rkcdayco7yh9"/></p><p>At this point you will be prompted to provide a key pair or create a new key pair.</p><p><img src="https://api.contentstack.io/v2/assets/576b084285453e6f4e3b4c90/download?uid=blt795f4d51585c445f" data-sys-asset-uid="blt795f4d51585c445f" alt="Select Key Pair.png" data-image="5je0e1g63tsa"/></p><p>This is necessary to enable SSH access to the EC2 instance. If you need help setting up a key pair the <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html" target="_blank">Amazon EC2 Key Pairs</a> article provides an overview and instructions for creating new new key pairs.</p><p>Start up the EC2 instances and take note of the assigned private IP addresses which we will use in a following step.</p><h2>Part II - Installing Elasticsearch</h2><h3>RPM</h3><p>Log into each EC2 instance via SSH.</p><pre class="prettyprint">$ ssh -v -i /pathto/[certfilename].pem ec2-user@[ec2hostname]<br /></pre><p>Then install the Elasticsearch RPM package on each EC2 instance as instructed below.</p><pre class="prettyprint">$ sudo rpm -i https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm<br /></pre><p>Other versions of Elasticsearch are available <a href="/downloads/elasticsearch">here</a>. Refer to <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.2/install-elasticsearch.html">the guide</a> if you prefer installing with yum.</p><p>Register Elasticsearch as a system service.</p><pre class="prettyprint">$ sudo chkconfig --add elasticsearch<br /></pre><h3>Install Plugins</h3><p>You need to install&nbsp;AWS cloud plugin on each EC2 instance in the cluster.</p><pre class="prettyprint">$ cd /usr/share/elasticsearch/ <br />$ sudo bin/plugin install cloud-aws<br /></pre><p>If there are any additional plugins you need, such as Marvel, for monitoring, or ICU, for additional language support, now is a good time to install them.</p><h3>Configure Elasticsearch</h3><p>The maximum JVM heap size should be based upon the machine's memory. Open "/etc/sysconfig/elasticsearch" on each EC2 instance with your favorite editor and set the "ES_HEAP_SIZE" and "MAX_LOCKED_MEMORY" parameters. The following configuration will fit a m3.2xlarge instance. The "ES_HEAP_SIZE" is recommended to be half of the memory but not more than 32GB.</p><pre class="prettyprint">ES_HEAP_SIZE=15g <br />MAX_LOCKED_MEMORY=unlimited<br /></pre><p>Open "/etc/elasticsearch/elasticsearch.yml" on every machine and edit the following settings.</p><pre class="prettyprint">cluster.name:esonaws <br />bootstrap.mlockall: true <br />discovery.zen.ping.unicast.hosts: [_ip_address_,…] <br />network.host: [_ip_address_]<br /></pre><p>"<strong>discovery.zen.ping.unicast.hosts</strong>" is a list of EC2 instance private IP addresses. All the master-eligible nodes must be listed. In a small cluster all nodes can be configured as both master nodes and data nodes. <strong>"network.host"</strong> is the EC2 instance private&nbsp;IP address&nbsp;of&nbsp;this host that is shared with the&nbsp;other nodes in the cluster.</p><p>The IP address is not required for a single node cluster. "_site_" and "_local_" represent the private address and the local loopback address "127.0.0.1" and allow access to those from remote.</p><h3>Starting Up and Verification</h3><p>If you are setting up multiple Elasticsearch nodes, they must all be the same version, same plugins and equivalent configurations. Start up Elasticsearch on each EC2 instance.</p><pre class="prettyprint">$ sudo service elasticsearch start<br /></pre><p>Once sta;d, let's verify the Elasticsearch cluster by using curl to request the cluster state.</p><pre class="prettyprint">$ curl localhost:9200/_cluster/health?pretty<br /></pre><p>If successful, the "status" will be "green" (or "yellow" for a single node). The "number_of_nodes" should be the same number of nodes started. Depending on your index settings,&nbsp;you will need a minimum of two nodes for the cluster "status" to turn "green". A minimum of 3 nodes is recommended to avoid leader election conflicts.&nbsp;If "status" isn't "green" you can also check the Security Group configuration or check logs under "/var/log/elasticsearch" for errors.</p><pre class="prettyprint">{ <br />  “cluster_name” : “esonaws”, <br />  “status” : “green”, <br />  “timed_out” : false, <br />  “number_of_nodes” : 3, <br />  “number_of_data_nodes” : 3, <br />  “active_primary_shards” : 8, <br />  “active_shards” : 16, <br />  “relocating_shards” : 0, <br />  “initializing_shards” : 0, <br />  “unassigned_shards” : 0, <br />  “delayed_unassigned_shards” : 0, <br />  “number_of_pending_tasks” : 0, <br />  “number_of_in_flight_fetch” : 0, <br />  “task_max_waiting_in_queue_millis” : 0, <br />  “active_shards_percent_as_number” : 100.0 <br />}<br /></pre><p>Your Elasticsearch cluster is ready!</p><h2>Summary</h2><p>Deploying an Elasticsearch cluster on Amazon EC2 is relatively easy, but it does require a number of configuration steps, familiarity with SSH, key pair management, and also assumes that you will be managing the machines.</p><p>If you prefer the ease-of-use of a managed service, <a href="/cloud">Elastic Cloud</a>, Elastic's official hosted Elasticsearch and Kibana offering on AWS, is a great choice. You can spin up a cluster in just a few clicks. <a href="/cloud">Elastic Cloud</a> also comes with Security, Kibana, supported Plugins, on-demand cluster scaling, automatic version backup and more.</p><p><img src="https://api.contentstack.io/v2/assets/576b08501f82392051463842/download?uid=blt16e32e38151bfbcb" data-sys-asset-uid="blt16e32e38151bfbcb" alt="elastic cloud is easy vi.gif" data-image="a8r3ea2idlwa"/></p><p>The <a href="/cloud/signup">Elastic Cloud trial</a> is free and doesn't require a credit card. Here's a link to a&nbsp;short video that describes <a href="https://www.youtube.com/watch?v=D743DdFmon0" target="_blank">Elastic Cloud</a> in a little more detail. You can also <a href="https://www.elastic.co/webinars/getting-started-with-elastic-cloud-with-gcp-and-aws" target="_self"><span>get started with Elastic Cloud on GCP and AWS</span></a>.</p>

Es-on-AWS-Blog-1.jpg

Es-on-AWS-Blog-720x420.jpg

Running Elasticsearch on AWS

<table style="background: #FFFFD2;">
<tbody>
<tr>
	<td><strong>Important note:</strong> Please see our documentation for updated information on <a href="/guide/en/elastic-stack-overview/current/xpack-ccr.html" style="color:#0077CC">cross-cluster replication capabilities</a> in the Elastic Stack.
	</td>
</tr>
</tbody>
</table><h3>Overview</h3><p>Organizations often produce and consume data in multiple regions, sometimes within a country and sometimes globally. Regions often have local data centers to meet local security, privacy or performance requirements. Data created in one region may need to be accessed in other regions. Requirements that may play a role in determining an appropriate architecture include high availability, fault tolerance, disaster recovery, ingestion latency and search and access latency. Limited network bandwidth and high latency between data centers can be key considerations for determining a multi-region and multi-data center architecture.
</p><p>Queueing data is a central theme in distributed architectures for reasons including guaranteed message delivery, low network bandwidth and message throughput variation. The purpose of this blog is to show and discuss proposed solutions to a use case where distributed departments produce data locally that must be replicated across data centers and be made accessible for search and access.
</p><h3>Use Case</h3><p>Assume a creative agency with offices in New York and London where media assets, such as images, videos, HTML documents, CSS documents, etc are created in the respective offices and maintained in separate Digital Asset Management (DAM) systems. Media assets are shared between New York and London and are discoverable through a search user interface.
</p><p><img src="https://api.contentstack.io/v2/uploads/598883ff8853622f0911f4f7/download?uid=blt4a926926851aef57" data-sys-asset-uid="blt4a926926851aef57" alt="img1.jpg">
</p><h3><strong>Requirements</strong></h3><strong><ul>
	<li dir="ltr">Media assets are created locally at both data centers and the data must be replicated to between data centers</li>
	<li dir="ltr">Both New York's and London's indices should be eventually consistent</li>
	<li dir="ltr">Search latency is critical as media authors need the ability to rapidly discover and repurpose existing assets</li>
	<li dir="ltr">Indexing latency, while important, isn't as critical and eventual consistency across data centers is acceptable</li>
</ul><h4>Considerations</h4><p>The blog post <a href="/blog/clustering_across_multiple_data_centers">Clustering Across Multiple Data Centers</a> addresses key concerns and provides guidance around the following topics:
</p><ul>
	<li dir="ltr">High Latency Between Data Centers</li>
	<li dir="ltr">Limited or Unreliable Connectivity</li>
	<li dir="ltr">Data Availability</li>
</ul><p>This post elaborates upon the guidance by incrementally refining architectures for the proposed use case.
</p><h3>Potential Architectures</h3><h4>Single Shared Elasticsearch Cluster Distributed Across Data Centers</h4><p>One might start by defining a single shared Elasticsearch cluster where nodes are distributed between the New York and London data centers. Such a configuration is discouraged; it’s a mistake you will regret. Elasticsearch clustering was designed assuming high speed and highly reliable networking. Elastic recommends clusters to be located in the same datacenter and rack, preferably. This assumes network failures in the same data center are infrequent whereas network failures between different geographic areas are more frequent. If the Elasticsearch minimum master nodes are configured incorrectly, a network failure or even a high latency network can result in a situation where master nodes are elected at both data centers, indices become unsynchronized and data is potentially lost.
</p><h4>Independent Elasticsearch Clusters Searchable by Tribe Nodes</h4><p>An alternative architecture is to have independent Elasticsearch clusters in each data center where each cluster indexes local media assets and&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-tribe.html">Tribe Nodes</a> federate searches across New York’s and London’s indices. There are a couple potential issues with this architecture. One being, if the network connection fails, then New York content authors won’t be able to search London assets and vice-versa. Another issue being the requirement to support rapid discovery will not be achieved in a high latency network environment; search response times will be determined by the slowest data center connection.
</p><h4>Independent Elasticsearch Clusters and A Shared Kafka Cluster</h4><p>In this architecture there are independent Elasticsearch clusters in each data center where each cluster indexes assets from a common Kafka queue which is shared across data centers. &nbsp;There is a single Kafka cluster, distributed between New York’s and London’s data centers. A concern with this architecture is that Kafka uses Zookeeper for cluster management.
</p><p>Similar to the concern of a minimum master nodes misconfiguration, provisioning a Zookeeper ensemble across data centers, especially in high latency environments, can result in partitioning; see&nbsp;<a href="https://zookeeper.apache.org/doc/r3.4.1/zookeeperObservers.html">ZooKeeper Observers</a>. If the two data centers are disconnected, or if there is a just some network lag, leader election may occur. In the data center where a quorum is determined and a leader is elected, read and write operations will continue. In the non-quorum data center, reads will continue, but writes will not be possible.
</p><h4>Independent Elasticsearch and Kafka Clusters</h4><p>Given the limitations of the other architectures, an alternative is to maintain two independent environments and to synchronize data between data centers. What is required is the ability to replicate data reliably between data centers. In this architecture, each data center operates independently, allowing authors to author content into and search a local Elasticsearch cluster having a complete data set. Changes that are made in the local data center are queued locally for remote consumption.
</p><p>There are two proposed architectures for replicating local changes to the remote data center. The first is to use Kafka MirrorMaker to replicate the queues to the remote data center. The second is to use Logstash to pull data from the remote data center’s queue.
</p><p>In the first proposed architecture the local queue is replicated to the remote (mirror) data center using MirrorMaker. A local instance of Logstash pulls data from the mirror of the remote Kafka Queue. MirrorMaker is a useful tool, having a small footprint, requiring no configuration and through a single command line request, MirrorMaker mirrors data between Kafka clusters. In this architecture local Logstash instances read from the local mirrored queues.
</p><p><img src="https://api.contentstack.io/v2/uploads/59888422d4a55d5c099ad498/download?uid=blt56091b764d11e002" data-sys-asset-uid="blt56091b764d11e002" alt="Multi Data Center MirrorMaker and Logstash">
</p><p>In the second proposed architecture a simplification is made and MirrorMaker is taken out of the picture as it isn’t needed. MirrorMaker doesn’t address feeding ElasticSearch; it serves purely as a queue replication technology. There may be cases where MirrorMaker is useful, for example, if there are consumers in addition to Logstash, but in this case Logstash is the only consumer. Logstash is able to pull data from the remote data center.
</p><p><img src="https://api.contentstack.io/v2/uploads/5988842f4558bdc02f8c3ed6/download?uid=blt4d3f03fcb8aa292d" data-sys-asset-uid="blt4d3f03fcb8aa292d" alt="Multi Data Center Logstash Pull Remote">
</p><p>There are additional benefits to having independent and decoupled installations. If there is a catastrophic failure at one location the other location remains fully operational. &nbsp;The operational location continues uninterrupted. And, if required, users of the failure site can access the remote location. Additionally, the fully operational location can serve as a backup to be restored to the failure location.
</p><h3>Summary</h3><p>While Elasticsearch currently doesn’t support reliable and eventually consistent replication across a high latency network, the combination of Elasticsearch and Kafka supports the configuration of an architecture that supports the proposed use case.
</p></strong>

data-centre.jpg

Scaling Elasticsearch Across Data Centers With Kafka

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

dara.png

Dara Gies

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

El contenido de esta página no está disponible en el idioma seleccionado. Elastic está trabajando para garantizar que el contenido esté disponible en varios idiomas. Gracias por tu paciencia mientras trabajamos.

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/elastic-cloud-account-terms" target="_self">Condiciones de servicio</a>, y que acepta recibir las <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">comunicaciones seleccionadas</a> en los detalles de contacto que proporcionó anteriormente. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/serverless-preview-terms" target="_self">Condiciones de servicio</a>, y que Elastic puede <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">ponerse en contacto con usted</a> acerca de nuestros productos y servicios relacionados. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>Al enviar esta información, aceptas los <a href="/es/agreements/global/cloud-services-monthly" rel="nofollow">Términos de servicio de Elastic</a>. Los datos personales se procesarán de conformidad con la <a href="/es/legal/privacy-statement">declaración de privacidad de Elastic</a>.
</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/terms-of-use" target="_self">Condiciones de servicio</a>, y que Elastic puede <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">ponerse en contacto con usted</a> acerca de nuestros productos y servicios relacionados. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.</p>

Hosted by

All Solutions

Follow us on Youtube

Watch now (no PT)

Articles by

Share by Email

Filters

All (no PT translation)

Contact information

See when this webinar starts in my time zone

Ver cuándo comienza este webinar en mi zona horaria

Thank you for your interest!

Register to Attend

Small image for

Register to Watch

Continue reading

Featured webinar

What to explore next...

Reset all

Load more press releases

Sign In to Attend

Share

Upcoming webinar

Learn more

Search Integrations

Share on Reddit

Share on Twitter

Share on Facebook

Load more news

View more posts

On-demand webinar

Follow us on Facebook

Author

Press Release

You'll also receive an email with related content.

You'll also receive an email with related content

Además, te enviaremos contenidos relevantes

Print

Contact Info

Overview

Video for

Read less

Register now

Share this story

See All Posts

See all top stories

View next

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

¿No puedes asistir? Regístrate y te enviaremos la grabación. También recibirás un correo electrónico con contenido relacionado.

Follow us on LinkedIn

Highlights

Explore similar demos

Follow us on Twitter

Share on LinkedIn

Load More

See more insights

Speakers

Close

Thank you for registering. We will send you a confirmation email soon.

Gracias por registrarte. Te enviaremos un correo electrónico de confirmación pronto.

Author

Articles by Dara Gies

Migrating to Elastic Cloud

Using Beats with Amazon AWS

Running Elasticsearch on AWS

Scaling Elasticsearch Across Data Centers With Kafka

Sign up for Elastic Cloud free trial