Getting started | Elasticsearch Plugins and Integrations [6.3]

» » »

« Google Cloud Storage Repository Plugin Client Settings »

Getting startededit

The plugin uses the Google Cloud Storage JSON API (v1) to connect to the Storage service. If this is the first time you use Google Cloud Storage, you first need to connect to the Google Cloud Platform Console and create a new project. Once your project is created, you must enable the Cloud Storage Service for your project.

Creating a Bucketedit

Google Cloud Storage service uses the concept of Bucket as a container for all the data. Buckets are usually created using the Google Cloud Platform Console. The plugin will not automatically create buckets.

To create a new bucket:

Connect to the Google Cloud Platform Console
Select your project
Go to the Storage Browser
Click the "Create Bucket" button
Enter the name of the new bucket
Select a storage class
Select a location
Click the "Create" button

The bucket should now be created.

Service Authenticationedit

The plugin supports two authentication modes:

The built-in Compute Engine authentication. This mode is recommended if your Elasticsearch node is running on a Compute Engine virtual machine.
Specifying Service Account credentials.

Using Compute Engineedit

When running on Compute Engine, the plugin use Google’s built-in authentication mechanism to authenticate on the Storage service. Compute Engine virtual machines are usually associated to a default service account. This service account can be found in the VM instance details in the Compute Engine console.

This is the default authentication mode and requires no configuration.

The Compute Engine VM must be allowed to use the Storage service. This can be done only at VM creation time, when "Storage" access can be configured to "Read/Write" permission. Check your instance details at the section "Cloud API access scopes".

Using a Service Accountedit

If your Elasticsearch node is not running on Compute Engine, or if you don’t want to use Google’s built-in authentication mechanism, you can authenticate on the Storage service using a Service Account file.

To create a service account file:

Connect to the Google Cloud Platform Console
Select your project
Got to the Permission tab
Select the Service Accounts tab
Click on "Create service account"
Once created, select the new service account and download a JSON key file

A service account file looks like this:

{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "service-account-for-your-repository@your-project-id.iam.gserviceaccount.com",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "..."
}

This file must be stored in the elasticsearch keystore, under a setting name of the form gcs.client.NAME.credentials_file, where NAME is the name of the client configuration. The default client name is default, but a different client name can be specified in repository settings using client.

For example, if specifying the credentials file in the keystore under gcs.client.my_alternate_client.credentials_file, you can configure a repository to use these credentials like this:

PUT _snapshot/my_gcs_repository
{
  "type": "gcs",
  "settings": {
    "bucket": "my_bucket",
    "client": "my_alternate_client"
  }
}