Monitor Google Cloud Platformedit

In this tutorial, you’ll learn how to monitor your Google Cloud Platform (GCP) deployments using Elastic Observability: Logs and Infrastructure metrics.

If you don’t want to provision VM and install data shippers due to process and management overhead, you can skip this step and ingest logs directly from Pub/Sub in the Google Cloud Console to Elastic with GCP Dataflow Templates.

What you’ll learnedit

You’ll learn how to:

Before you beginedit

Create a deployment using our hosted Elasticsearch Service on Elastic Cloud. The deployment includes an Elasticsearch cluster for storing and searching your data, and Kibana for visualizing and managing your data. For more information, see Spin up the Elastic Stack.

Step 1: Setup a Service Accountedit

Google Cloud Platform implements service accounts as a way to access APIs securely. To monitor GCP with Elastic, you will need a service account. The easiest way is to use a predefined service account that GCP creates automatically. Alternatively, you can create a new service account. This tutorial creates a new one.

First, to access the service account menu, click MenuIAM & AdminService Accounts.

Service account menu

Next, click Create Service Account. Define the new service account name (for example, "gcp-monitor") and the description (for example, "Service account to monitor GCP services using the Elastic Stack").

Service account name

Make sure to select the correct roles.

To monitor GCP services, you need to add these roles to the service account:

Compute Viewer:

Service account roles compute viewer

Monitoring Viewer:

Service account roles monitoring viewer

Pub/Sub Subscriber:

Service account roles pub/sub subscriber

The final result should be the following:

Service account roles result

Click Continue, then skip granting users access to this service. Finally, click Done. The service account is now ready to be used.

Next, to use the service account, click Manage keys.

Service account manage keys

Then, add a new JSON key type by selecting Create new key.

Service account create key

After that, the credential file is downloaded. Keep this file in an accessible place to use later.

Step 2: Install and configure Metricbeatedit

This tutorial assumes the Elastic cluster is already running. Make sure you have your cloud ID and your credentials on hand.

To monitor GCP using the Elastic Stack, you need two main components: an Elastic deployment to store and analyze the data and an agent to collect and ship the data.

Two agents can be used to monitor GCP: Metricbeat is used to monitor metrics, and Filebeat to monitor logs. You can run the agents on any machine. This tutorial uses a small GCP instance, e2-small (2 vCPUs, 2 GB memory), with an Ubuntu distribution.

Install Metricbeatedit

Download and install Metricbeat.

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.4.2-amd64.deb
sudo dpkg -i metricbeat-8.4.2-amd64.deb

Set up assetsedit

Metricbeat comes with predefined assets for parsing, indexing, and visualizing your data. Run the following command to load these assets. It may take a few minutes.

./metricbeat setup -e -E 'cloud.id=YOUR_DEPLOYMENT_CLOUD_ID' -E 'cloud.auth=elastic:YOUR_SUPER_SECRET_PASS' 

Substitute your Cloud ID and an administrator’s username:password in this command. To find your Cloud ID, click on your deployment.

Setting up Metricbeat is an admin-level task that requires extra privileges. As a best practice, use an administrator role to set up, and a more restrictive role for event publishing (which you will do next).

Configure Metricbeat outputedit

Next, you are going to configure Metricbeat output to Elasticsearch Service.

  1. Use the Metricbeat keystore to store secure settings. Store the Cloud ID in the keystore.

    ./metricbeat keystore create
    echo -n "<Your Deployment Cloud ID>" | ./metricbeat keystore add CLOUD_ID --stdin
  2. To store metrics in Elasticsearch with minimal permissions, create an API key to send data from Metricbeat to Elasticsearch Service. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and select ManagementDev Tools. Send the following request:

    POST /_security/api_key
    {
      "name": "metricbeat-monitor",
      "role_descriptors": {
        "metricbeat_writer": {
          "cluster": ["monitor", "read_ilm"],
          "index": [
            {
              "names": ["metricbeat-*"],
              "privileges": ["view_index_metadata", "create_doc"]
            }
          ]
        }
      }
    }
  3. The response contains an api_key and an id field, which can be stored in the Metricbeat keystore in the following format: id:api_key.

    echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./metricbeat keystore add ES_API_KEY --stdin

    Make sure you specify the -n parameter; otherwise, you will have painful debugging sessions due to adding a newline at the end of your API key.

  4. To see if both settings have been stored, run the following command:

    ./metricbeat keystore list
  5. To configure Metricbeat to output to Elasticsearch Service, edit the metricbeat.yml configuration file. Add the following lines to the end of the file.

    cloud.id: ${CLOUD_ID}
    output.elasticsearch:
      api_key: ${ES_API_KEY}
  6. Finally, test if the configuration is working. If it is not working, verify if you used the right credentials and add them again.

    ./metricbeat test output

Now that the output is working, you are going to set up the input (GCP).

Step 3: Configure Metricbeat Google Cloud Platform moduleedit

To collect metrics from Google Cloud Platform, use the Google Cloud Platform module. This module periodically fetches monitoring metrics from Google Cloud Platform using Stackdriver Monitoring API for Google Cloud Platform services.

Extra GCP charges on Stackdriver Monitoring API requests may be generated by this module. Please see rough estimation of the number of API calls for more details.

  1. Enable the GCP module.

    ./metricbeat modules enable gcp
  2. Edit the modules.d/gcp.yml file to configure which metrics to collect.

    - module: gcp
      metricsets:
        - compute 
      zone: "" 
      project_id: "your-project-id" 
      period: 1m 
      credentials_file_path: "/home/ubuntu/credentials.json" 

    The compute metricset is a predefined metricset that collects some GCP compute metrics.

    Defines which zones to monitor, an empty value collects data from all zones

    Collects metrics within the your-project-id project-id.

    Collects metrics every minute

    The GCP credential file that you generated earlier. (Don’t forget to create the file if it does not exist and use the correct full path).

  3. To check if Metricbeat can collect data, test the input by running the following command:

    ./metricbeat test modules gcp

    Metricbeat will print GCP metrics to the terminal, if the setup is correct.

  4. When the input and output are ready, start Metricbeat to collect the data.

    ./metricbeat -e
  5. Finally, log into Kibana and open the [Metricbeat GCP] Compute Overview dashboard.

    Metricbeat compute overview dashboard

Step 4: Install and configure Filebeatedit

Now that Metricbeat is up and running, configure Filebeat to collect Google Cloud logs.

Install Filebeatedit

Download and install Filebeat.

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.4.2-amd64.deb
sudo dpkg -i filebeat-8.4.2-amd64.deb

Set up assetsedit

Filebeat comes with predefined assets for parsing, indexing, and visualizing your data. Run the following command to load these assets. It may take a few minutes.

./filebeat setup -e -E 'cloud.id=YOUR_DEPLOYMENT_CLOUD_ID' -E 'cloud.auth=elastic:YOUR_SUPER_SECRET_PASS' 

Substitute your Cloud ID and an administrator’s username:password in this command. To find your Cloud ID, click on your deployment.

Setting up Filebeat is an admin-level task that requires extra privileges. As a best practice, use an administrator role to set up and a more restrictive role for event publishing (which you will do next).

Configure Filebeat outputedit

Next, you are going to configure Filebeat output to Elasticsearch Service.

  1. Use the Filebeat keystore to store secure settings. Store the Cloud ID in the keystore.

    ./filebeat keystore create
    echo -n "<Your Deployment Cloud ID>" | ./filebeat keystore add CLOUD_ID --stdin
  2. To store logs in Elasticsearch with minimal permissions, create an API key to send data from Filebeat to Elasticsearch Service. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and select ManagementDev Tools. Send the following request:

    POST /_security/api_key
    {
      "name": "filebeat-monitor-gcp",
      "role_descriptors": {
        "filebeat_writer": {
          "cluster": [
            "monitor",
            "read_ilm",
            "cluster:admin/ingest/pipeline/get", 
            "cluster:admin/ingest/pipeline/put" 
          ],
          "index": [
            {
              "names": ["filebeat-*"],
              "privileges": ["view_index_metadata", "create_doc"]
            }
          ]
        }
      }
    }

    Filebeat needs extra cluster permissions to publish logs, which differs from the Metricbeat configuration. You can find more details here.

  3. The response contains an api_key and an id field, which can be stored in the Filebeat keystore in the following format: id:api_key.

    echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./filebeat keystore add ES_API_KEY --stdin

    Make sure you specify the -n parameter; otherwise, you will have painful debugging sessions due to adding a newline at the end of your API key.

  4. To see if both settings have been stored, run the following command:

    ./filebeat keystore list
  5. To configure Filebeat to output to Elasticsearch Service, edit the filebeat.yml configuration file. Add the following lines to the end of the file.

    cloud.id: ${CLOUD_ID}
    output.elasticsearch:
      api_key: ${ES_API_KEY}
  6. Finally, test if the configuration is working. If it is not working, verify that you used the right credentials and, if necessary, add them again.

    ./filebeat test output

Now that the output is working, you are going to set up the input (GCP).

Step 5: Configure Filebeat Google Cloud moduleedit

To collect logs from Google Cloud Platform, use the Google Cloud Platform module. This module periodically fetches logs that have been exported from Stackdriver to a Google Pub/Sub topic sink. There are three available filesets: audit, vpcflow, firewall. This tutorial covers the audit fileset.

  1. Go to the Logs Router page to configure GCP to export logs to a Pub/Sub topic. Use the search bar to find the page:

    Navigate to Logs Router page

    To set up the logs routing sink, click Create sink. Set sink name as monitor-gcp-audit-sink. Select the Cloud Pub/Sub topic as the sink service and Create new Cloud Pub/Sub topic named monitor-gcp-audit:

    Create Pub/Sub topic

    Finally, under Choose logs to include in sink, add logName:"cloudaudit.googleapis.com" (it includes all audit logs). Click create sink. It will look something like the following:

    Create logs routing sink
  2. Now go to the Pub/Sub page to add a subscription to the topic you just created. Use the search bar to find the page:

    GCP Pub/Sub

    To add a subscription to the monitor-gcp-audit topic click Create subscription:

    Create GCP Pub/Sub Subscription

    Set monitor-gcp-audit-sub as the Subscription ID and leave the Delivery type as pull:

    GCP Pub/Sub Subscription ID

    Finally, scroll down and click Create.

  3. Now that GCP is configured to export audit logs, enable Filebeat Google Cloud module.

    ./filebeat modules enable gcp
  4. Edit the modules.d/gcp.yml file with the following configurations.

    - module: gcp
      vpcflow:
        enabled: false 
      firewall:
        enabled: false 
      audit:
        enabled: true 
        var.project_id: "elastic-education" 
        var.topic: "monitor-gcp-audit" 
        var.subscription_name: "monitor-gcp-audit-sub" 
        var.credentials_file: "/home/ubuntu/credentials.json" 

    Disables both vpcflow and firewall filesets.

    Enables the audit fileset.

    Collects data within the elastic-education project-id.

    Collects logs from the monitor-gcp-audit topic.

    Google Cloud Pub/Sub topic subscription name.

    The GCP credential file that you generated earlier. (Don’t forget to create the file if it does not exist and to use the correct full path).

  5. Start Filebeat to collect the logs.

    ./filebeat -e
  6. Finally, log into Kibana and open the [Filebeat GCP] Audit dashboard.

    Filebeat audit overview dashboard