Snapshotting to Google Cloud Storage (GCS) for Elasticsearch 6.x

Snapshotting to GCS isn’t currently supported in the Cloud UI, but there is a workaround using Application Default Credentials (https://developers.google.com/identity/protocols/application-default-credentials). This method uses keys built in to your GCP instances to login to GCS.

To use the Application Default Credentials it is necessary to make a small change to the Instance Template when you create your GCP instances Google Cloud Platform Configuration.

Note

This method only works for Elasticsearch clusters at version 6.x. If you are using an earlier Elasticsearch version you must upgrade first.

  1. Create a Google instance template with access to the Storage API:

    When creating an Instance Template, under Identity and API access you need to set either: Allow full access to all Cloud APIs or Set access for each API and select Read Write under Storage

    Creating an instance template with default auth
  2. Log into the Cloud UI.
  3. Add the snapshot repository configuration through the Advanced Cluster Configuration:

    1. Create a deployment with an Elasticsearch cluster that has the repository-gcs plugin enabled.
    2. From your deployment menu, go to the Edit page then go to the bottom of the page and click advanced Elasticsearch configuration.
    3. In the Data text area, carefully edit the JSON to add a top-level gcp_resources configuration object with your bucket name for snapshotting (test-ece-auth in this case). Also set enabled under snapshot to true.:
GCP resources cluster data
  1. Press the Data section Save button to store the updated cluster data and trigger a plan change.

The cluster should then have snapshots enabled and will begin snapshotting either immediately (if skip_snapshot is false in the plan) or in 30 minutes (the default snapshot interval).