Elastic Observability monitors metrics for Google Cloud in just minutes

By

Jonathan Simon,

Eric Lowry

serverless-launch-blog-image.jpg

Developers and SREs choose to host their applications on Google Cloud Platform (GCP) for its reliability, speed, and ease of use. On Google Cloud, development teams are finding additional value in migrating to Kubernetes on GKE, leveraging the latest serverless options like Cloud Run, and improving traditional, tiered applications with managed services.

Elastic Observability offers 16 out-of-the-box integrations for Google Cloud services with more on the way. A full list of Google Cloud integrations can be found in our online documentation.

In addition to our native Google Cloud integrations, Elastic Observability aggregates not only logs but also metrics for Google Cloud services and the applications running on Google Cloud compute services (Compute Engine, Cloud Run, Cloud Functions, Kubernetes Engine). All this data can be analyzed visually and more intuitively using Elastic®’s advanced machine learning (ML) capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML based metrics correlations, read: APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

That’s right, Elastic offers metrics ingest, aggregation, and analysis for Google Cloud services and applications on Google Cloud compute services. Elastic is more than logs — it offers a unified observability solution for Google Cloud environments.

In this blog, I’ll review how Elastic Observability can monitor metrics for a three-tier web application running on Google Cloud services, which include:

  • Google Cloud Run
  • Google Cloud SQL for PostgreSQL
  • Google Cloud Memorystore for Redis
  • Google Cloud VPC Network

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start reviewing metrics.

Prerequisites and config

Here are some of the components and details we used to set up this demonstration:

  • Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
  • Ensure you have a Google Cloud project and a Service Account with permissions to pull the necessary data from Google Cloud (see details in our documentation).
  • We used Google Cloud’s three-tier app and deployed it using the Google Cloud console.
  • We’ll walk through installing the general Elastic Google Cloud Platform Integration, which covers the services we want to collect metrics for.
  • We will not cover application monitoring; instead, we will focus on how Google Cloud services can be easily monitored.
  • In order to see metrics, you will need to load the application. We’ve also created a playwright script to drive traffic to the application.

Three-tier application overview

Before we dive into the Elastic configuration, let's review what we are monitoring. If you follow the Jump Start Solution: Three-tier web app instructions for deploying the task-tracking app, you will have the following deployed.

1

What’s deployed:

  • Cloud Run frontend tier that renders an HTML client in the user's browser and enables user requests to be sent to the task-tracking app
  • Cloud Run middle tier API layer that communicates with the frontend and the database tier
  • Memorystore for Redis instance in the database tier, caching and serving data that is read frequently
  • Cloud SQL for PostgreSQL instance in the database tier, handling requests that can't be served from the in-memory Redis cache

At the end of the blog, we will also provide a Playwright script that can be run to send requests to this app in order to load it with example data and exercise its functionality. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to get the application, Google Cloud integration on Elastic, and what gets ingested.

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

2 - start free trial

Step 1: Deploy the Google Cloud three-tier application

Follow the instructions listed out in Jump Start Solution: Three-tier web app choosing the Deploy through the console option for deployment.

Step 2: Create a Google Cloud Service Account and download credentials file

Once you’ve installed the app, the next step is to create a Service Account with a Role and a Service Account Key that will be used by Elastic’s integration to access data in your Google Cloud project.

Go to Google Cloud IAM Roles to create a Role with the necessary permissions. Click the CREATE ROLE button.

3

Give the Role a Title and an ID. Then add the 10 assigned permissions listed here.  

  • cloudsql.instances.list
  • compute.instances.list
  • monitoring.metricDescriptors.list
  • monitoring.timeSeries.list
  • pubsub.subscriptions.consume
  • pubsub.subscriptions.create
  • pubsub.subscriptions.get
  • pubsub.topics.attachSubscription
  • redis.instances.list
  • run.services.list

These permissions are a minimal set of what’s required for this blog post. You should add permissions for all the services for which you would like to collect metrics. If you need to add or remove permissions in the future, the Role’s permissions can be updated as many times as necessary.

Click the CREATE button.

4

Go to Google Cloud IAM Service Accounts to create a Service Account that will be used by the Elastic integration for access to Google Cloud. Click the CREATE SERVICE ACCOUNT button.

5

Enter a Service account name and a Service account ID. Click the CREATE AND CONTINUE button.

6

Then select the Role that you created previously and click the CONTINUE button.

7

Click the DONE button to complete the Service Account creation process.

Next select the Service Account you just created to see its details page. Under the KEYS tab, click the ADD KEY dropdown and select Create new key.

8

In the Create private key dialog window, with the Key type set as JSON, click the CREATE button.

9

The JSON credentials file key will be automatically downloaded to your local computer’s Downloads folder. The credentials file will be named something like:

your-project-id-12a1234b1234.json

You can rename the file to be something else. For the purpose of this blog, we’ll rename it to:

credentials.json

Step 3: Create a Google Cloud VM instance

To create the Compute Engine VM instance in Google Cloud, go to Compute Engine. Then select CREATE INSTANCE.

10

Enter the following values for the VM instance details:

  • Enter a Name of your choice for the VM instance. 
  • Expand the Advanced Options section and the Networking sub-section.
    • Enter allow-ssh as the Networking tag. 
    • Select the Network Interface to use the tiered-web-app-private-network, which is the network on which the Google Cloud three-tier web app is deployed.

Click the CREATE button to create the VM instance.

11

Step 4: SSH in to the Google Cloud VM instance and upload the credentials file

In order to SSH into the Google Cloud VM instance you just created in the previous step, you’ll need to create a Firewall rule in tiered-web-app-private-network, which is the network where the VM instance resides.

Go to the Google Cloud Firewall policies page. Click the CREATE FIREWALL RULE button.

12

Enter the following values for the Firewall Rule.

  • Enter a firewall rule Name. 
  • Select tiered-web-app-private-network for the Network. 
  • Enter allow-ssh for Target Tags.
  • Enter 0.0.0.0/0 for the Source IPv4 ranges.Click TCP and set the Ports to 22.

Click CREATE to create the firewall rule.

13

After the new Firewall rule is created, you can now SSH into your VM instance. Go to the Google Cloud VM instances and select the VM instance you created in the previous step to see its details page. Click the SSH button.

14

Once you are SSH’d inside the VM instance terminal window, click the UPLOAD FILE button.

15

Select the credentials.json file located on your local computer and click the Upload Files button to upload the file.

16

In the VM instance’s SSH terminal, run the following command to get the full path to your Google Cloud Service Account credentials file.

realpath credentials.json

This should return the full path to your Google Cloud Service Account credentials file.

17

Copy the credentials file’s full path and save it in a handy location to be used in a later step.

Step 5: Add the Elastic Google Cloud integration

Navigate to the Google Cloud Platform integration in Elastic by selecting Integrations from the top-level menu. Search for google and click the Google Cloud Platform tile.

18

Click Add Google Cloud Platform.

19

Click Add integration only (skip agent installation).

20

Update the Project Id input text box to be your Google Cloud Project ID. Next, paste in the credentials file’s full path into the Credentials File input text box.

21

As you can see, the general Elastic Google Cloud Platform Integration will collect a significant amount of data from 16 Google Cloud services. If you don’t want to install this general Elastic Google Cloud Platform Integration, you can select individual integrations to install. Click Save and continue.

You’ll be presented with a confirmation dialog window. Click Add Elastic Agent to your hosts.

22

This will display the instructions required to install the Elastic agent. Copy the command under the Linux Tar tab.

23

Next you will need to use SSH to log in to the Google Cloud VM instance and run the commands copied from Linux Tar tab. Go to Compute Engine. Then click the name of the VM instance that you created in Step 2. Log in to the VM by clicking the SSH button.

24 - instance

Once you are SSH’d inside the VM instance terminal window, run the commands copied previously from Linux Tar tab in the Install Elastic Agent on your host instructions.

When the installation completes, you’ll see a confirmation message in the Install Elastic Agent on your host form. Click the Add the integration button.

25 - add agent

Excellent! The Elastic agent is sending data to Elastic Cloud. Now let’s observe some metrics.

Step 6: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic and exercise the functionality of the Google Cloud three-tier application:

import { test, expect } from '@playwright/test';

test('homepage for Google Cloud Threetierapp', async ({ page }) => {
  await page.goto('https://tiered-web-app-fe-zg62dali3a-uc.a.run.app');
  // Insert 2 todo items
  await page.fill('id=todo-new', (Math.random()*100).toString());
  await page.keyboard.press('Enter');
  await page.waitForTimeout(1000);
  await page.fill('id=todo-new', (Math.random()*100).toString());
  await page.keyboard.press('Enter');
  await page.waitForTimeout(1000);
  // Click one todo item
  await page.getByRole('checkbox').nth(0).check();
  await page.waitForTimeout(1000);
  // Delete one todo item
  const deleteButton = page.getByText('delete').nth(0);
  await deleteButton.dispatchEvent('click');
  await page.waitForTimeout(4000);
});

Step 7: Go to Google Cloud dashboards in Elastic

With Elastic Agent running, you can go to Elastic Dashboards to view what’s being ingested. Simply search for “dashboard” in Elastic and choose Dashboards.

26 - dashboard

This will open the Elastic Dashboards page.

27

In the Dashboards search box, search for GCP and click the [Metrics GCP] CloudSQL PostgreSQL Overview dashboard, one of the many out-of-the-box dashboards available. Let’s see what comes up.

28

On the Cloud SQL dashboard, we can see the following sampling of some of the many available metrics:

  • Disk write ops
  • CPU utilization
  • Network sent and received bytes
  • Transaction count
  • Disk bytes used
  • Disk quota
  • Memory usage
  • Disk read ops

Next let’s take a look at metrics for Cloud Run.

29 - line graphs

We’ve created a custom dashboard using the Create dashboard button on the Elastic Dashboards page. Here we see a few of the numerous available metrics:

  • Container instance count
  • CPU utilization for the three-tier app frontend and API
  • Request count for the three-tier app frontend and API
  • Bytes in and out of the API
30

This is a custom dashboard created for MemoryStore where we can see the following sampling of the available metrics:

  • Network traffic to the Memorystore Redis instance
  • Count of the keys stored in Memorystore Redis
  • CPU utilization of the Memorystore Redis instance
  • Memory usage of the Memorystore Redis instance

Congratulations, you have now started monitoring metrics from key Google Cloud services for your application!

What to monitor on Google Cloud next?

Add logs from Google Cloud Services

Now that metrics are being monitored, you can also now add logging. There are several options for ingesting logs.

The Google Cloud Platform Integration in the Elastic Agent has four separate logs settings: audit logs, firewall logs, VPC Flow logs, and DNS logs. Just ensure you turn on what you wish to receive.

31

Analyze your data with Elastic machine learning

Once metrics and logs (or either one) are in Elastic, start analyzing your data through Elastic’s ML capabilities. A great review of these features can be found here:

Conclusion: Monitoring Google Cloud service metrics with Elastic Observability is easy!

I hope you’ve gotten an appreciation for how Elastic Observability can help you monitor Google Cloud service metrics. Here’s a quick recap of lessons and what you learned:

  • Elastic Observability supports ingest and analysis of Google Cloud service metrics.
  • It’s easy to set up ingest from Google Cloud services via the Elastic Agent.
  • Elastic Observability has multiple out-of-the-box Google Cloud service dashboards you can use to preliminarily review information and then modify for your needs.
  • For metrics not covered by out-of-the-box dashboards, custom dashboards can be easily created to visualize metrics that are important to you.
  • 16 Google Cloud services are supported as part of Google Cloud Platform Integration on Elastic Observability, with more services being added regularly.
  • As noted in related blogs, you can analyze your Google Cloud service metrics with Elastic’s machine learning capabilities.

Try it out for yourself by signing up via Google Cloud Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on Google Cloud around the world. Your Google Cloud Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with Google Cloud.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial