Elastic Observability offers 16 out-of-the-box integrations for Google Cloud services with more on the way. A full list of Google Cloud integrations can be found in our online documentation.

In addition to our native Google Cloud integrations, Elastic Observability aggregates not only logs but also metrics for Google Cloud services and the applications running on Google Cloud compute services (Compute Engine, Cloud Run, Cloud Functions, Kubernetes Engine). All this data can be analyzed visually and more intuitively using Elastic®’s advanced machine learning (ML) capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML based metrics correlations, read: APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

That’s right, Elastic offers metrics ingest, aggregation, and analysis for Google Cloud services and applications on Google Cloud compute services. Elastic is more than logs — it offers a unified observability solution for Google Cloud environments.

In this blog, I’ll review how Elastic Observability can monitor metrics for a three-tier web application running on Google Cloud services, which include:

• Google Cloud Run
• Google Cloud SQL for PostgreSQL
• Google Cloud Memorystore for Redis
• Google Cloud VPC Network

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start reviewing metrics.

Prerequisites and config

Here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• Ensure you have a Google Cloud project and a Service Account with permissions to pull the necessary data from Google Cloud (see details in our documentation).
• We used Google Cloud’s three-tier app and deployed it using the Google Cloud console.
• We’ll walk through installing the general Elastic Google Cloud Platform Integration, which covers the services we want to collect metrics for.
• We will not cover application monitoring; instead, we will focus on how Google Cloud services can be easily monitored.
• In order to see metrics, you will need to load the application. We’ve also created a playwright script to drive traffic to the application.

Three-tier application overview

Before we dive into the Elastic configuration, let's review what we are monitoring. If you follow the Jump Start Solution: Three-tier web app instructions fordeploying the task-tracking app, you will have the following deployed.

What’s deployed:

• Cloud Run frontend tier that renders an HTML client in the user's browser and enables user requests to be sent to the task-tracking app
• Cloud Run middle tier API layer that communicates with the frontend and the database tier
• Memorystore for Redis instance in the database tier, caching and serving data that is read frequently
• Cloud SQL for PostgreSQL instance in the database tier, handling requests that can't be served from the in-memory Redis cache

At the end of the blog, we will also provide a Playwright script that can be run to send requests to this app in order to load it with example data and exercise its functionality. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to get the application, Google Cloud integration on Elastic, and what gets ingested.

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Deploy the Google Cloud three-tier application

Follow the instructions listed out in Jump Start Solution: Three-tier web app choosing the Deploy through the console option for deployment.

Step 2: Create a Google Cloud Service Account and download credentials file

Once you’ve installed the app, the next step is to create a Service Account with a Role and a Service Account Key that will be used by Elastic’s integration to access data in your Google Cloud project.

Go to Google Cloud IAM Roles to create a Role with the necessary permissions. Click the CREATE ROLE button.

Give the Role a Title and an ID. Then add the 10 assigned permissions listed here.

• cloudsql.instances.list
• compute.instances.list
• monitoring.metricDescriptors.list
• monitoring.timeSeries.list
• pubsub.subscriptions.consume
• pubsub.subscriptions.create
• pubsub.subscriptions.get
• pubsub.topics.attachSubscription
• redis.instances.list
• run.services.list

These permissions are a minimal set of what’s required for this blog post. You should add permissions for all the services for which you would like to collect metrics. If you need to add or remove permissions in the future, the Role’s permissions can be updated as many times as necessary.

Click the CREATE button.

Go to Google Cloud IAM Service Accounts to create a Service Account that will be used by the Elastic integration for access to Google Cloud. Click the CREATE SERVICE ACCOUNT button.

Enter a Service account name and a Service account ID. Click the CREATE AND CONTINUE button.

Then select the Role that you created previously and click the CONTINUE button.

Click the DONE button to complete the Service Account creation process.

Next select the Service Account you just created to see its details page. Under the KEYS tab, click the ADD KEY dropdown and select Create new key.

In the Create private key dialog window, with the Key type set as JSON, click the CREATE button.

The JSON credentials file key will be automatically downloaded to your local computer’s Downloads folder. The credentials file will be named something like:

your-project-id-12a1234b1234.json

You can rename the file to be something else. For the purpose of this blog, we’ll rename it to:

credentials.json

Step 3: Create a Google Cloud VM instance

To create the Compute Engine VM instance in Google Cloud, go to Compute Engine. Then select CREATE INSTANCE.

Enter the following values for the VM instance details:

• Enter a Name of your choice for the VM instance.
• Expand the Advanced Options section and the Networking sub-section.
  • Enter allow-ssh as the Networking tag.
  • Select the Network Interface to use the tiered-web-app-private-network , which is the network on which the Google Cloud three-tier web app is deployed.

Click the CREATE button to create the VM instance.

Step 4: SSH in to the Google Cloud VM instance and upload the credentials file

In order to SSH into the Google Cloud VM instance you just created in the previous step, you’ll need to create a Firewall rule in tiered-web-app-private-network , which is the network where the VM instance resides.

Go to the Google Cloud Firewall policies page. Click the CREATE FIREWALL RULE button.

Enter the following values for the Firewall Rule.

• Enter a firewall rule Name.
• Select tiered-web-app-private-network for the Network.
• Enter allow-ssh for Target Tags.
• Enter 0.0.0.0/0 for the Source IPv4 ranges.Click TCP and set the Ports to 22.

Click CREATE to create the firewall rule.

After the new Firewall rule is created, you can now SSH into your VM instance. Go to the Google Cloud VM instances and select the VM instance you created in the previous step to see its details page. Click the SSH button.

Once you are SSH’d inside the VM instance terminal window, click the UPLOAD FILE button.

Select the credentials.json file located on your local computer and click the Upload Files button to upload the file.

In the VM instance’s SSH terminal, run the following command to get the full path to your Google Cloud Service Account credentials file.

realpath credentials.json

This should return the full path to your Google Cloud Service Account credentials file.

Copy the credentials file’s full path and save it in a handy location to be used in a later step.

Step 5: Add the Elastic Google Cloud integration

Navigate to the Google Cloud Platform integration in Elastic by selecting Integrations from the top-level menu. Search for google and click the Google Cloud Platform tile.

Click Add Google Cloud Platform.

Click Add integration only (skip agent installation).

Update the Project Id input text box to be your Google Cloud Project ID. Next, paste in the credentials file’s full path into the Credentials File input text box.

As you can see, the general Elastic Google Cloud Platform Integration will collect a significant amount of data from 16 Google Cloud services. If you don’t want to install this general Elastic Google Cloud Platform Integration, you can select individual integrations to install. Click Save and continue.

You’ll be presented with a confirmation dialog window. Click Add Elastic Agent to your hosts.

This will display the instructions required to install the Elastic agent. Copy the command under the Linux Tar tab.

Next you will need to use SSH to log in to the Google Cloud VM instance and run the commands copied from Linux Tar tab. Go to Compute Engine. Then click the name of the VM instance that you created in Step 2. Log in to the VM by clicking the SSH button.

Once you are SSH’d inside the VM instance terminal window, run the commands copied previously from Linux Tar tab in the Install Elastic Agent on your host instructions.

When the installation completes, you’ll see a confirmation message in the Install Elastic Agent on your host form. Click the Add the integration button.

Excellent! The Elastic agent is sending data to Elastic Cloud. Now let’s observe some metrics.

Step 6: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic and exercise the functionality of the Google Cloud three-tier application:

import { test, expect } from "@playwright/test";

test("homepage for Google Cloud Threetierapp", async ({ page }) => {
  await page.goto("https://tiered-web-app-fe-zg62dali3a-uc.a.run.app");
  // Insert 2 todo items
  await page.fill("id=todo-new", (Math.random() * 100).toString());
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=todo-new", (Math.random() * 100).toString());
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  // Click one todo item
  await page.getByRole("checkbox").nth(0).check();
  await page.waitForTimeout(1000);
  // Delete one todo item
  const deleteButton = page.getByText("delete").nth(0);
  await deleteButton.dispatchEvent("click");
  await page.waitForTimeout(4000);
});

Step 7: Go to Google Cloud dashboards in Elastic

With Elastic Agent running, you can go to Elastic Dashboards to view what’s being ingested. Simply search for “dashboard” in Elastic and choose Dashboards.

This will open the Elastic Dashboards page.

In the Dashboards search box, search for GCP and click the [Metrics GCP] CloudSQL PostgreSQL Overview dashboard, one of the many out-of-the-box dashboards available. Let’s see what comes up.

On the Cloud SQL dashboard, we can see the following sampling of some of the many available metrics:

• Disk write ops
• CPU utilization
• Network sent and received bytes
• Transaction count
• Disk bytes used
• Disk quota
• Memory usage
• Disk read ops

Next let’s take a look at metrics for Cloud Run.

We’ve created a custom dashboard using the Create dashboard button on the Elastic Dashboards page. Here we see a few of the numerous available metrics:

• Container instance count
• CPU utilization for the three-tier app frontend and API
• Request count for the three-tier app frontend and API
• Bytes in and out of the API

This is a custom dashboard created for MemoryStore where we can see the following sampling of the available metrics:

• Network traffic to the Memorystore Redis instance
• Count of the keys stored in Memorystore Redis
• CPU utilization of the Memorystore Redis instance
• Memory usage of the Memorystore Redis instance

Congratulations, you have now started monitoring metrics from key Google Cloud services for your application!

What to monitor on Google Cloud next?

Add logs from Google Cloud Services

Now that metrics are being monitored, you can also now add logging. There are several options for ingesting logs.

The Google Cloud Platform Integration in the Elastic Agent has four separate logs settings: audit logs, firewall logs, VPC Flow logs, and DNS logs. Just ensure you turn on what you wish to receive.

Analyze your data with Elastic machine learning

Once metrics and logs (or either one) are in Elastic, start analyzing your data through Elastic’s ML capabilities. A great review of these features can be found here:

• Correlating APM Telemetry to determine root causes in transactions
• Introduction to Elastic Machine Learning

Conclusion: Monitoring Google Cloud service metrics with Elastic Observability is easy!

I hope you’ve gotten an appreciation for how Elastic Observability can help you monitor Google Cloud service metrics. Here’s a quick recap of lessons and what you learned:

• Elastic Observability supports ingest and analysis of Google Cloud service metrics.
• It’s easy to set up ingest from Google Cloud services via the Elastic Agent.
• Elastic Observability has multiple out-of-the-box Google Cloud service dashboards you can use to preliminarily review information and then modify for your needs.
• For metrics not covered by out-of-the-box dashboards, custom dashboards can be easily created to visualize metrics that are important to you.
• 16 Google Cloud services are supported as part of Google Cloud Platform Integration on Elastic Observability, with more services being added regularly.
• As noted in related blogs, you can analyze your Google Cloud service metrics with Elastic’s machine learning capabilities.

Try it out for yourself by signing up via Google Cloud Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on Google Cloud around the world. Your Google Cloud Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with Google Cloud.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic is committed to supporting Google Kubernetes Engine (GKE) in all of its delivery modes. In October, during the Google Cloud Next ‘22 event, we announced our intention to integrate and certify Elastic Agent on Anthos, Autopilot, Google Distributed Cloud, and more.

Since that event, we have worked together with Google to get the Elastic Agent certified for use on Anthos, but we didn’t stop there.

Today we are happy to announce that we have been certified for operation on GKE Autopilot.

Hands on with Elastic and GKE Autopilot

Kubernetes observability has never been easier

To show how easy it is to get started with Autopilot and Elastic, let's walk through deploying the Elastic Agent on an Autopilot cluster. I’ll show how easy it is to set up and monitor an Autopilot cluster with the Elastic Agent and observe the cluster’s behavior with Kibana integrations.

One of the main differences between GKE and GKE Autopilot is that Autopilot protects the system namespace “kube-system.” To increase the stability and security of a cluster, Autopilot prevents user space workloads from adding or modifying system pods. The default configuration for Elastic Agent is to install itself into the system namespace. The majority of the changes we will make here are to convince the Elastic Agent to run in a different namespace.

Let’s get started with Elastic Stack!

While writing this article, I used the latest version of Elastic. The best way for you to get started with Elastic Observability is to:

1. Get an account on Elastic Cloud and look at this tutorial to help launch your first stack, or
2. Launch Elastic Cloud on your Google Account

Provisioning an Autopilot cluster and an Elastic stack

To test the agent, I first deployed the recommended, default GKE Autopilot cluster. Elastic’s GKE integration supports kube-state-metrics (KSM), which will increase the number of reported metrics available for reporting and dashboards. Like the Elastic Agent, KSM defaults to running in the system namespace, so I modified its manifest to work with Autopilot. For my testing, I also deployed a basic Elastic stack on Elastic Cloud in the same Google region as my Autopilot cluster. I used a fresh cluster deployed on Elastic’s managed service (ESS), but the process is the same if you are using an Elastic Cloud subscription purchased through the Google marketplace.

Adding Elastic Observability to GKE Autopilot

Because this is a brand new deployment, Elastic suggests adding integrations to it. Let’s add the Kubernetes integration into the new deployment:

Elastic offers hundreds of integrations; filter the list by typing “kub” into the search bar (1) and then click the Kubernetes integration (2).

The Kubernetes integration page gives you an overview of the integration and lets you manage the Kubernetes clusters you want to observe. We haven’t added a cluster yet, so I clicked “Add Kubernetes” to add the first integration.

I changed the integration name to reflect the Kubernetes offering type and then clicked “Save and continue” to accept the integration defaults.

At this point, an Agent policy has been created. Now it’s time to install the agent. I clicked on the “Kubernetes” integration.

Then I selected the “integration policies” tab (1) and clicked “Add agent” (2).

Finally, I downloaded the full manifest for a standard GKE environment.

We won’t be using this manifest directly, but it contains many of the values that we will need to deploy the agent on Autopilot in the next section.

The Elastic stack is ready and waiting for the Autopilot logs, metrics, and events. It’s time to connect Autopilot to this deployment using the Elastic Agent for GKE.

Connect Autopilot to Elastic

From the Google cloud terminal, I downloaded and edited the Elastic Agent manifest for GKE Autopilot.

$ curl -o elastic-agent-managed-gke-autopilot.yaml \
https://github.com/elastic/elastic-agent/blob/autopilotdocumentaton/docs/manifests/elastic-agent-managed-gke-autopilot.yaml

I used the cloud shell editor to configure the manifest for my Autopilot and Elastic clusters. For example, I updated the following:

containers:
  - name: elastic-agent
    image: docker.elastic.co/beats/elastic-agent:8.19.13

I also changed the agent to the version of Elastic that I installed (8.6.0).

From the Integration manifest I downloaded earlier, I copied the values for FLEET_URL and FLEET_ENROLLMENT_TOKEN into this YAML file.

Now it’s time to apply the updated manifest to the Autopilot instance.

Before I commit, I always like to see what’s going to be created (and check for syntax errors) with a dry run.

$ clear
$ kubectl apply --dry-run="client" -f elastic-agent-managed-gke-autopilot.yaml

Everything looks good, so I’ll do it for real this time.

$ clear
$ kubectl apply -f elastic-agent-managed-gke-autopilot.yaml

After several minutes, metrics will start flowing from the Autopilot cluster directly into the Elastic deployment.

Adding a workload to the Autopilot cluster

Observing an Autopilot cluster without a workload is boring, so I deployed a modified version of Google’s Hipster Shop (which includes OpenTelemetry reporting):

$ git clone https://github.com/bshetti/opentelemetry-microservices-demo
$ cd opentelemetry-microservices-demo
$ nano ./deploy-with-collector-k8s/otelcollector.yaml

To get the application’s telemetry talking to our Elastic stack, I replaced all instances of the exporter type from HTTP (otlphttp/elastic) to gRPC (otlp/elastic). I then replaced OTEL_EXPORTER_OTLP_ENDPOINT with my APM endpoint and I replaced OTEL_EXPORTER_OTLP_HEADERS with my APM OTEL Bearer and Token.

Then I deployed the Hipster Shop.

$ kubectl create -f ./deploy-with-collector-k8s/adservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/redis.yaml
$ kubectl create -f ./deploy-with-collector-k8s/cartservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/checkoutservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/currencyservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/emailservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/frontend.yaml
$ kubectl create -f ./deploy-with-collector-k8s/paymentservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/productcatalogservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/recommendationservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/shippingservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/loadgenerator.yaml

Once all of the shop’s pods were running, I deployed the OpenTelemetry collector.

$ kubectl create -f ./deploy-with-collector-k8s/otelcollector.yaml

Observe and visualize Autopilot’s metrics

Now that we have added the Elastic Agent to our Autopilot cluster and added a workload, let's take a look at some of the Kubernetes visualizations the integration provides out of the box.

The “[Metrics Kubernetes] Overview” is a great place to start. It provides a high-level view of the resources used by the cluster and allows me to drill into more specific dashboards that I find interesting:

For example, the “[Metrics Kubernetes] Pods” gives me a high-level view of the pods deployed in the cluster:

The “[Metrics Kubernetes] Volumes” gives me an in-depth view to how storage is allocated and used in the Autopilot cluster:

Creating an alert

From here, I can easily discover patterns in my cluster’s behavior and even create Alerts. Here is an example of an alert to notify me if the the main storage volume (called “volume”) exceeds 80% of its allocated space:

With a little work, I created this view from the standard dashboard:

Conclusion

Today I have shown how easy it is to monitor, observe, and generate alerts on a GKE Autopilot cluster. To get more information on what is possible, see the official Elastic documentation for Autopilot observability with Elastic Agent.

Next steps

If you don’t have Elastic yet, you can get started for free with an Elastic Trial today. Get more from Elastic and Google together with a Marketplace subscription. Elastic does more than just integrate with GKE — check out the almost 300 integrations that Elastic provides.