10 Steps for automating Elastic workload migration from on-prem to Elastic on AWS

By

Udayasimha Theepireddy (Uday),

Battulga Purevragchaa

blog-thumb-cloud-aws.png

Are you ready to migrate your self-managed Elastic workloads to Elastic on Amazon Web Services (AWS) to leverage cost, time and scale efficiencies? Take advantage of the best practices and tools we developed with AWS to ensure your fast and smooth transition. Let’s run through your self-service migration path options, then go over the simple “how-to” steps to automate your migration.

Choose your path

The three main options to add your on-premises data sets to Elastic in your AWS environment are data re-ingestion, snapshot and restore, and reindexing from a cluster.

1. If your data source (applications, databases and infrastructure) is also going to be migrated to AWS, then the data can then be re-ingested into Elastic from your source. Elastic provides integrations with popular AWS services to help streamline data ingestion — all you have to do is click to capture, store and search data.

2. Snapshot and restore is ideal when data in existing clusters need to be moved to Elastic Cloud, and the existing cluster is running an earlier Elasticsearch version than Elastic Cloud.

3. The other option is reindex from a cluster. This is best suited for when your data volumes are low and/or there are constraints to using a snapshot and restore, and the existing cluster is running an earlier Elasticsearch version than Elastic Cloud. Find more details on how to use the Reindex API in the migration guide.

In this blog, we will provide the simple steps to automate your migration path with snapshot and restore.

Automate your migration with snapshot and restore

Let’s go through how easy it is for you to migrate your data using snapshot and restore with Terraform (open-source infrastructure as code software). First, let’s begin with the prerequisites and then we will cover the how-to steps.

Prerequisites

  1. Verify that the target Elastic Cloud is running a version that is the same or higher than the current Elasticsearch clusters. A more detailed table of version compatibility is listed in the Elastic snapshot and restore guide.
  2. Check to confirm that the infrastructure capacity of the Elastic Cloud cluster is sufficient to accommodate the data volume and computing requirements for the use case.
  3. Check the listed limitations in the Elastic snapshot and restore guide, and version-specific breaking changes in the Elastic upgrade guide, to confirm that there are no constraints that affect the deployment’s migration to Elastic Cloud.
  4. Check the correct version of Terraform software installed and also that the Terraform vault feature is enabled.
  5. Gather necessary information such as Elasticsearch API key (var.apikey) and AWS access key (var.s3_client_access_key, var.s3_client_secret_key).
  6. Check network connectivity (direct connect) between on-premises and Elastic on AWS VPC and other AWS resource permissions.
  7. If you do not already have the Elasticsearch AWS CLI and Amazon Simple Storage Service (Amazon S3) plugin on your existing Elasticsearch cluster, you need to install it from the Elasticsearch home directory with the following command:
> sudo bin/elasticsearch-plugin install repository-s3
  1. Add the Amazon S3 access keys to the Elasticsearch keystore with the following commands from the root directory of the existing Elasticsearch cluster (enter the appropriate keys from the earlier Amazon S3 bucket creation process, when prompted).
> bin/elasticsearch-keystore add s3.client.default.access_key
> bin/elasticsearch-keystore add s3.client.default.secret_key
Elastic Cloud Migration Architecture

Migration steps

  1. Create and register an Elastic Cloud snapshot repository by using the Amazon S3 service.
  2. Create and configure a local snapshot repository and point to the Amazon S3 bucket.
  3. Create a new snapshot from the local cluster and store it in the Amazon S3 bucket.
  4. Provision the Elastic Cloud cluster.
  5. Provision AWS resources needed for the Elastic Agent.
  6. Provision the Elastic serverless forwarder needed for frictionless log ingestion.
  7. Create multiple AWS Identity and Access Management (IAM) instance roles with fine-grained permissions for access to AWS services necessary for the deployment process.
  8. Close all indices in Elastic Cloud.
  9. Restore the local cluster data from the snapshot in Elastic Cloud.
  10. Finally, open all indices in Elastic Cloud.

Deployment steps

Below are the detailed steps which were briefly mentioned in the prerequisites section such as gathering necessary secret access keys. If you already have your keys, you can skip to step 3.

  1. Generate an Elasticsearch Service (ESS) API Key:
    1. Open your browser and navigate to https://cloud.elastic.co/login.
    2. Log in with your email and password.
    3. Click on Elasticsearch Service.
    4. Navigate to Features > API Keys and click on Generate API Key.
    5. Choose a name for your API key.
    6. Save your API key somewhere safe.
  2. Create an AWS access key:
    1. Create a S3 bucket policy that allows List, Read, Write permissions.
    2. Create an IAM user and attach the policy created above.
    3. Ensure to check Programmatic access for the IAM user.
    4. Save your AWS access key somewhere safe.
  3. Clone your Terraform Elastic Cloud migration repository using the following commands:
    1. git clone https://github.com/aws-ia/terraform-elastic-cloud
    2. cd terraform-elastic-cloud
  4. If you are using HashiCorp Vault, you can set the Vault environment variables as follows:
    1. export VAULT_ADDR= "Your Vault URL"
    2. export VAULT_TOKEN="Your Vault Token.                                                                                  Add your Elasticsearch API Key (we used “ess” for the path as an example):
vault kv put secret/ess apikey="Your Elastic API Key"

Add your AWS API Key (we used “aws” for the path as an example):

vault kv put secret/aws s3_client_access_key="Your AWS Access Key"     
s3_client_secret_key="Your AWS Secret Key"
  1. You need to provide the keys — which can be done in a number of ways in Terraform. We will use Variable Definitions (.tfvars). This newly added feature automates your migration of your Elasticsearch cluster to Elastic Cloud. Files are shown below (if you are using HashiCorp Vault).

All you need to do is to assign the URL of your self-managed Elasticsearch like the example below.

    var.local_elasticsearch_url (e.g., http://127.0.0.1:9200).

    For example, we can add var.local_elasticsearch_url in the same testing.tfvars as shown below (if you are using HashiCorp Vault):

    name = "Elasticsearch Cluster"
local_elasticsearch_url = ""
vault_address = "Your Local Elastic Cluster URL"
vault_ess_path = "secret/ess"
vault_aws_path = "secret/aws"
apikey = "hashicorp/vault"
s3_client_access_key = "hashicorp/vault"
s3_client_secret_key  = "hashicorp/vault"

    Alternatively, you can directly provide keys as follows:

    name = "Elasticsearch Cluster"
local_elasticsearch_url = ""
apikey = "Your Elastic API Key"
s3_client_access_key = "Your AWS Access Key"
s3_client_secret_key  = "Your AWS Secret Key"
    1. Once these variables are specified in .tfvars file (in the example below, we are using a file named testing.tfvars), you can run the Terraform module as follows:
          terraform init
terraform validate
terraform apply -var-file="testing.tfvars"

          Destroy Deployment

          If you no longer need the infrastructure that are provisioned by the terraform module, you can easily terminate all resources with the following command:
          terraform destroy -var-file="testing.tfvars"

          Leverage more migration resources with Elastic and AWS

          Elastic has been certified as an AWS ISV Workload Migration Program (WMP) partner to support the migration of self-managed on-premises Elasticsearch or ELK workloads to Elastic on AWS. Elastic’s methodology provides customers with a prescriptive and repeatable approach for accelerating migration adoption and is directly aligned with migration guidance and best practices published by AWS, including the AWS Well-Architected Framework and AWS Migration Acceleration Program. Through the AWS WMP program, you can further take advantage of onboarding guidance, migration resources, and AWS infrastructure credits and funding assistance to achieve your business goals and accelerate your cloud journey.

          Get started today on your migration to Elastic on AWS

          Migrate to take advantage of all the cloud-enabled features, using the Elastic solutions your team already knows. View this migration guide, quickstart guide, website and documentation, or reach out to us directly to find out more about how Elastic can help with your migration — including assessment, planning and financial assistance. You can also reach out to our Elastic Consulting team that can help you migrate complex and large scale workloads. Take advantage of a 7-day free trial directly from the AWS Marketplace and deploy Elastic in minutes.

          The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.


          Battulga Purevragchaa

          Battulga Purevragchaa is a Partner Solutions Architect with AWS ISV Workload Migration Program. He helps AWS partners to build migration accelerators for their customers leveraging his years of industry and cloud experience.

          Sign up for Elastic Cloud free trial

          Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

          Start free trial