Configure a snapshot repository using GCSedit

Configure a custom snapshot repository using your Google Cloud Storage account.

Set up your service account credentialsedit

You’ll need to have an existing Google Cloud account and have the appropriate permissions to generate credentials.

  1. Create a service account key in your Google Cloud project.

    The service account should be granted the role of storage.admin so that Elasticsearch clusters can read, write, and list the bucket objects.

  2. Save the service account key in JSON file format. You are going to use it later to configure your Elasticsearch deployment for snapshotting.

For more detailed information on the JSON account service key, see Using a Service Account.

Prepare a bucketedit

Follow the Google Cloud Storage documentation to create a GCS bucket. The bucket must be in the same region as your deployment.

Enable the repository-gcs pluginedit

To access your GCS bucket, your deployment must have the repository-gcs plugin enabled. On Google Cloud Platform, the plugin is enabled by default. If your deployment is on AWS or Azure, follow these steps to enable the repository-gcs plugin:

  1. See Google Cloud Storage Repository Plugin to download the version of the plugin that matches your Elastic Stack version.
  2. Upload the plugin to your deployment:

    1. Log in to the Elasticsearch Add-On for Heroku console.
    2. Open the Custom plugins page and click Add plugin.
    3. Specify the plugin name (repository-gcs) and the version.
    4. Select An installable plugin (Compiled, no source code).
    5. Click Create plugin.
    6. Navigate back to the Custom plugins page.
    7. Click the plugin name.
    8. Drag and drop to upload the repository-gcs plugin zip file.

Configure the keystoreedit

Create an entry for the GCS client in the Elasticsearch keystore:

  1. Log in to the Elasticsearch Add-On for Heroku console.
  2. Navigate to the Security page of the deployment you wish to configure.
  3. Click Create settings.
  4. Enter the Setting name gcs.client.default.credentials_file.
  5. With Type set to JSON block / file, add your GCS service account key JSON file.
  6. Click Save.

Create the repositoryedit

For deployments with Elastic Stack version 7.2 and higher:

  1. Open Kibana and go to Management > Snapshot and Restore.
  2. On the Repositories tab, click Register a repository.
  3. Provide a name for your repository and select type Google Cloud Storage.
  4. Provide the following settings:

    • Client: default
    • Bucket: Your GCS bucket name
    • Base path: A directory to contain the snapshots

      • This setting is optional. Include a base_path if you have multiple clusters writing to the same GCS repository. This ensures that a snapshot won’t overwrite the snapshot metadata for another cluster.
  5. Add any other settings that you wish to configure.
  6. Click Register.
  7. Click Verify to confirm that your settings are correct and the deployment can connect to your repository.
  8. Your snapshot repository is now set up using GCS! You can use Kibana to manage your snapshots and begin sending Elasticsearch snapshots to your own bucket. For details see the Kibana Snapshot and Restore documentation.

For deployments with Elastic Stack version 7.1 and lower:

With the bucket ready, you can create a repository with a PUT request to /_snapshot/repository_name:

PUT _snapshot/repository_name
{
  "type": "gcs",
  "settings": {
    "bucket": "name-of-gcs-bucket",
    "compress": true
  }
}