To run ECF for GCP confidently at scale, you need to understand its capacity characteristics and sizing behavior. For ECF for GCP which is part of the broader ECF architecture, we answered those questions through repeatable load testing and by grounding decisions in measured data.

We'll introduce the test setup, explain each runtime setting, and share the capacity numbers we observed for a single instance.

How we load tested EDOT Cloud Forwarder for GCP

Architecture

The load testing architecture simulates a realistic, high-volume pipeline:

1. We developed a load tester service that uploads generated log files to a GCS bucket as fast as possible.
2. Each file creation in this Google Cloud Storage (GCS) bucket then triggers an event notification to Pub/Sub.
3. Pub/Sub delivers push messages to a Cloud Run service where EDOT Cloud Forwarder fetches and processes these log files.

Our setup exposes two primary tunable settings that directly influence Cloud Run scaling behavior and memory pressure:

• Request pressure using a concurrency setting (how many concurrent requests each ECF instance can handle).
• Work per request using a log count setting (number of logs per file in each uploaded object).

In our tests, we used a testing system that:

• Deploys the whole testing infrastructure. This includes the complete ECF infrastructure, a mock backend, etc.
• Generates log files according to the configured log counts, using a Cloud Audit log of ~1.4 KB.
• Runs a matrix of tests across all combinations of concurrency and log volume.
• Produces a report for each tested concurrency level in which several stats are reported, such as CPU usage and memory consumption.

For reproducibility and isolation, the otlphttp exporter in EDOT Cloud Forwarder uses a mock backend that always returns HTTP 200. This ensures all observed behavior is attributable to ECF itself, not downstream systems or network variability.

Step 1: Establish a stable runtime before measuring capacity

Before asking how much load a single instance can handle, we first established a stable runtime baseline.

We quickly learned that a single flag, cpu_idle, can turn Cloud Run into a garbage-collector (GC) starvation trap. This is amplified by a known limitation of our ECF current architecture: the existing OpenTelemetry implementation reads whole log files into memory before processing them. Our goal was to eliminate configuration side effects so capacity tests reflected ECF actual limits.

We focused on three runtime parameters:

All parameter-isolation tests use a single Cloud Run instance (min 0, max 1), fix concurrency for the scenario under study, and keep input files and test matrix identical across runs. This design lets us attribute differences directly to the parameter in question.

CPU allocation: Stop starving the garbage collector

Cloud Run offers two CPU allocation modes:

• Request-based (throttled). Enabled with cpu_idle: true. CPU is available only while a request is actively being processed.
• Instance-based (always on). Enabled with cpu_idle: false. CPU remains available when idle, allowing background work such as garbage collection to run.

The tests compared these modes under identical conditions:

What we observed

With CPU allocated only on requests (cpu_idle: true):

• Memory variance was extreme (±71% RSS, ±213% heap).
• Peak heap reached ~304 MB in the worst run.
• We saw request refusals in the sample (90% success rate).

With CPU always allocated (cpu_idle: false):

• Memory variance became tightly bounded (±8% RSS, ±32% heap).
• Peak heap dropped to ~89 MB in the worst run.
• We saw no refusals in the sample (100% success).

From these runs we saw:

• When CPU is throttled, the Go garbage collector is effectively starved, leading to heap accumulation and large run-to-run variance.
• When CPU is always available, garbage collection keeps pace with allocation, resulting in lower and more predictable memory usage.

Takeaway: for this set of tests, cpu_idle: false was the most stable baseline configuration. Request-based CPU throttling introduced artificial instability that makes capacity planning much harder.

Go memory limit: GOMEMLIMIT in constrained containers

Cloud Run enforces a hard memory limit at the container level. If the process exceeds it, the instance is OOM-killed.

We tested Cloud Run with:

The tests compared:

• No GOMEMLIMIT (Go relies on OS pressure).
• GOMEMLIMIT=460MiB (or 90% of container memory).

The results were clear:

Takeaway: in a memory-constrained environment like Cloud Run, setting GOMEMLIMIT close to (but below) the container limit is essential for predictable behavior under load.

GOGC: memory savings vs. reliability

The GOGC parameter controls how much the heap can grow (in %) between GC cycles:

• Lower values (e.g., GOGC=50): more frequent collections, lower memory, higher CPU.
• Higher values (e.g., GOGC=100): fewer collections, higher memory, lower CPU.

The tests covered: (1) GOGC=50 (aggressive); (2) GOGC=75 (moderate); (3) GOGC=100 (default/unset).

Setup:

What we observed

From the runs:

The conclusion from these runs is clear: pushing GOGC down trades memory for reliability, and the trade is not favorable for ECF.

Takeaway: for this workload, the default GOGC=100 provided the best balance. Attempts to optimize memory by lowering GOGC directly reduced reliability.

Step 2: Find capacity and breaking points

With the runtime stabilized, we evaluated how much traffic a single instance can sustain by increasing concurrency until failures emerged.

How to read the tables: each concurrency level was tested across 20 runs covering both light (240 logs per file, around 362KB file size) and heavy inputs (over 6k logs per file, around 8MB file size). Tables report baseline RSS from light workloads and peak values from the worst-case run.

Concurrency 5: Stable baseline

At concurrency 5, the service was solid.

This proved that a single instance handles a moderate load comfortably, with memory usage staying well within safe limits.

Concurrency 10: Safe but volatile

At concurrency 10, the system remained functional but with significant volatility.

We also noticed that memory usage shows extreme variance:

• Best run RSS: 178 MB.
• Worst run RSS: 425 MB.

This behavior comes mainly from two effects:

• Bursty Pub/Sub delivery: 10 heavy requests may land at nearly the same instant.
• The use of io.ReadAll inside the collector: each request reads the entire log file into memory.

When all 10 requests arrived concurrently, we were effectively stacking ~10× file size in RAM before the GC can clean up. When they are slightly staggered, GC has time to reclaim memory between requests, leading to much lower peaks.

This leads to a crucial sizing insight:

• Do not size the service using average memory (for example, ~260 MB).
• Size it for the worst observed burst (~425 MB) to avoid OOM or GC stalls.

In practice, you should set the memory limit to at least 512 MiB per instance at concurrency 10.

Concurrency 20: Unstable, systemic load shedding

At concurrency 20, the system consistently began shedding load.

Even though memory and CPU metrics don't look drastically worse than at concurrency 10, behavior changes qualitatively: the service begins to refuse requests consistently.

Concurrency 40: Failure mode

At concurrency 40, the instance collapsed completely. Memory and CPU are overwhelmed, and ingest reliability collapses.

The breaking point: a 1 vCPU instance's realistic limits

Combined with the CPU data (94% peak at concurrency 10), this supports a practical rule: for this workload and architecture, 10 concurrent heavy requests per 1 vCPU instance is the realistic upper bound.

Turning findings into concrete recommendations

These experiments lead to clear, actionable recommendations for running the ECF OpenTelemetry collector on Cloud Run as part of the broader Elastic Cloud Forwarder deployment.

Scope: these recommendations apply to the workload and harness we tested (light vs. heavy log files up to 8MB, and Pub/Sub burst delivery), using the tuned runtime settings listed below. If your log sizes, request burstiness, or pipeline shape differ significantly, validate these limits against your own traffic.

Runtime and container configuration

Capacity and per-instance limits

For a 1 vCPU Cloud Run instance running the ECF OpenTelemetry collector with the tuned runtime:

Scaling strategy: horizontal, not vertical

• Vertical scaling (increasing concurrency per instance) quickly runs into CPU and memory limits for this workload.
• Horizontal scaling is a better fit: treat each instance as a worker with a hard limit of 10 concurrent heavy jobs.

Practically:

• Configure the service so that no instance exceeds 10 concurrent requests.
• Let autoscaling handle an increased load by adding instances, not by increasing per-instance concurrency.

Takeaways

• Tuned runtime settings matter as much as raw resources: a single flag like cpu_idle can be the difference between predictable behavior and GC-driven chaos.
• Go needs explicit limits in containers: GOMEMLIMIT must be set in memory-constrained environments; otherwise, OOM kills are inevitable under heavy ingesting.
• "Lower memory" is not always better: aggressive GC tuning (GOGC < 100) did reduce memory usage but directly increased failure rates.
• Concurrency 10 is the realistic ceiling for a 1 vCPU ECF instance; beyond that, refusals and instability become the norm.
• Horizontal scaling is the right model: each instance should be treated as a 10-request worker, with higher total throughput coming from more workers rather than more concurrency per worker.

Elastic Observability offers 16 out-of-the-box integrations for Google Cloud services with more on the way. A full list of Google Cloud integrations can be found in our online documentation.

In addition to our native Google Cloud integrations, Elastic Observability aggregates not only logs but also metrics for Google Cloud services and the applications running on Google Cloud compute services (Compute Engine, Cloud Run, Cloud Functions, Kubernetes Engine). All this data can be analyzed visually and more intuitively using Elastic®’s advanced machine learning (ML) capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML based metrics correlations, read: APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

That’s right, Elastic offers metrics ingest, aggregation, and analysis for Google Cloud services and applications on Google Cloud compute services. Elastic is more than logs — it offers a unified observability solution for Google Cloud environments.

In this blog, I’ll review how Elastic Observability can monitor metrics for a three-tier web application running on Google Cloud services, which include:

• Google Cloud Run
• Google Cloud SQL for PostgreSQL
• Google Cloud Memorystore for Redis
• Google Cloud VPC Network

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start reviewing metrics.

Prerequisites and config

Here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• Ensure you have a Google Cloud project and a Service Account with permissions to pull the necessary data from Google Cloud (see details in our documentation).
• We used Google Cloud’s three-tier app and deployed it using the Google Cloud console.
• We’ll walk through installing the general Elastic Google Cloud Platform Integration, which covers the services we want to collect metrics for.
• We will not cover application monitoring; instead, we will focus on how Google Cloud services can be easily monitored.
• In order to see metrics, you will need to load the application. We’ve also created a playwright script to drive traffic to the application.

Three-tier application overview

Before we dive into the Elastic configuration, let's review what we are monitoring. If you follow the Jump Start Solution: Three-tier web app instructions fordeploying the task-tracking app, you will have the following deployed.

What’s deployed:

• Cloud Run frontend tier that renders an HTML client in the user's browser and enables user requests to be sent to the task-tracking app
• Cloud Run middle tier API layer that communicates with the frontend and the database tier
• Memorystore for Redis instance in the database tier, caching and serving data that is read frequently
• Cloud SQL for PostgreSQL instance in the database tier, handling requests that can't be served from the in-memory Redis cache

At the end of the blog, we will also provide a Playwright script that can be run to send requests to this app in order to load it with example data and exercise its functionality. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to get the application, Google Cloud integration on Elastic, and what gets ingested.

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Deploy the Google Cloud three-tier application

Follow the instructions listed out in Jump Start Solution: Three-tier web app choosing the Deploy through the console option for deployment.

Step 2: Create a Google Cloud Service Account and download credentials file

Once you’ve installed the app, the next step is to create a Service Account with a Role and a Service Account Key that will be used by Elastic’s integration to access data in your Google Cloud project.

Go to Google Cloud IAM Roles to create a Role with the necessary permissions. Click the CREATE ROLE button.

Give the Role a Title and an ID. Then add the 10 assigned permissions listed here.

• cloudsql.instances.list
• compute.instances.list
• monitoring.metricDescriptors.list
• monitoring.timeSeries.list
• pubsub.subscriptions.consume
• pubsub.subscriptions.create
• pubsub.subscriptions.get
• pubsub.topics.attachSubscription
• redis.instances.list
• run.services.list

These permissions are a minimal set of what’s required for this blog post. You should add permissions for all the services for which you would like to collect metrics. If you need to add or remove permissions in the future, the Role’s permissions can be updated as many times as necessary.

Click the CREATE button.

Go to Google Cloud IAM Service Accounts to create a Service Account that will be used by the Elastic integration for access to Google Cloud. Click the CREATE SERVICE ACCOUNT button.

Enter a Service account name and a Service account ID. Click the CREATE AND CONTINUE button.

Then select the Role that you created previously and click the CONTINUE button.

Click the DONE button to complete the Service Account creation process.

Next select the Service Account you just created to see its details page. Under the KEYS tab, click the ADD KEY dropdown and select Create new key.

In the Create private key dialog window, with the Key type set as JSON, click the CREATE button.

The JSON credentials file key will be automatically downloaded to your local computer’s Downloads folder. The credentials file will be named something like:

your-project-id-12a1234b1234.json

You can rename the file to be something else. For the purpose of this blog, we’ll rename it to:

credentials.json

Step 3: Create a Google Cloud VM instance

To create the Compute Engine VM instance in Google Cloud, go to Compute Engine. Then select CREATE INSTANCE.

Enter the following values for the VM instance details:

• Enter a Name of your choice for the VM instance.
• Expand the Advanced Options section and the Networking sub-section.
  • Enter allow-ssh as the Networking tag.
  • Select the Network Interface to use the tiered-web-app-private-network , which is the network on which the Google Cloud three-tier web app is deployed.

Click the CREATE button to create the VM instance.

Step 4: SSH in to the Google Cloud VM instance and upload the credentials file

In order to SSH into the Google Cloud VM instance you just created in the previous step, you’ll need to create a Firewall rule in tiered-web-app-private-network , which is the network where the VM instance resides.

Go to the Google Cloud Firewall policies page. Click the CREATE FIREWALL RULE button.

Enter the following values for the Firewall Rule.

• Enter a firewall rule Name.
• Select tiered-web-app-private-network for the Network.
• Enter allow-ssh for Target Tags.
• Enter 0.0.0.0/0 for the Source IPv4 ranges.Click TCP and set the Ports to 22.

Click CREATE to create the firewall rule.

After the new Firewall rule is created, you can now SSH into your VM instance. Go to the Google Cloud VM instances and select the VM instance you created in the previous step to see its details page. Click the SSH button.

Once you are SSH’d inside the VM instance terminal window, click the UPLOAD FILE button.

Select the credentials.json file located on your local computer and click the Upload Files button to upload the file.

In the VM instance’s SSH terminal, run the following command to get the full path to your Google Cloud Service Account credentials file.

realpath credentials.json

This should return the full path to your Google Cloud Service Account credentials file.

Copy the credentials file’s full path and save it in a handy location to be used in a later step.

Step 5: Add the Elastic Google Cloud integration

Navigate to the Google Cloud Platform integration in Elastic by selecting Integrations from the top-level menu. Search for google and click the Google Cloud Platform tile.

Click Add Google Cloud Platform.

Click Add integration only (skip agent installation).

Update the Project Id input text box to be your Google Cloud Project ID. Next, paste in the credentials file’s full path into the Credentials File input text box.

As you can see, the general Elastic Google Cloud Platform Integration will collect a significant amount of data from 16 Google Cloud services. If you don’t want to install this general Elastic Google Cloud Platform Integration, you can select individual integrations to install. Click Save and continue.

You’ll be presented with a confirmation dialog window. Click Add Elastic Agent to your hosts.

This will display the instructions required to install the Elastic agent. Copy the command under the Linux Tar tab.

Next you will need to use SSH to log in to the Google Cloud VM instance and run the commands copied from Linux Tar tab. Go to Compute Engine. Then click the name of the VM instance that you created in Step 2. Log in to the VM by clicking the SSH button.

Once you are SSH’d inside the VM instance terminal window, run the commands copied previously from Linux Tar tab in the Install Elastic Agent on your host instructions.

When the installation completes, you’ll see a confirmation message in the Install Elastic Agent on your host form. Click the Add the integration button.

Excellent! The Elastic agent is sending data to Elastic Cloud. Now let’s observe some metrics.

Step 6: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic and exercise the functionality of the Google Cloud three-tier application:

import { test, expect } from "@playwright/test";

test("homepage for Google Cloud Threetierapp", async ({ page }) => {
  await page.goto("https://tiered-web-app-fe-zg62dali3a-uc.a.run.app");
  // Insert 2 todo items
  await page.fill("id=todo-new", (Math.random() * 100).toString());
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=todo-new", (Math.random() * 100).toString());
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  // Click one todo item
  await page.getByRole("checkbox").nth(0).check();
  await page.waitForTimeout(1000);
  // Delete one todo item
  const deleteButton = page.getByText("delete").nth(0);
  await deleteButton.dispatchEvent("click");
  await page.waitForTimeout(4000);
});

Step 7: Go to Google Cloud dashboards in Elastic

With Elastic Agent running, you can go to Elastic Dashboards to view what’s being ingested. Simply search for “dashboard” in Elastic and choose Dashboards.

This will open the Elastic Dashboards page.

In the Dashboards search box, search for GCP and click the [Metrics GCP] CloudSQL PostgreSQL Overview dashboard, one of the many out-of-the-box dashboards available. Let’s see what comes up.

On the Cloud SQL dashboard, we can see the following sampling of some of the many available metrics:

• Disk write ops
• CPU utilization
• Network sent and received bytes
• Transaction count
• Disk bytes used
• Disk quota
• Memory usage
• Disk read ops

Next let’s take a look at metrics for Cloud Run.

We’ve created a custom dashboard using the Create dashboard button on the Elastic Dashboards page. Here we see a few of the numerous available metrics:

• Container instance count
• CPU utilization for the three-tier app frontend and API
• Request count for the three-tier app frontend and API
• Bytes in and out of the API

This is a custom dashboard created for MemoryStore where we can see the following sampling of the available metrics:

• Network traffic to the Memorystore Redis instance
• Count of the keys stored in Memorystore Redis
• CPU utilization of the Memorystore Redis instance
• Memory usage of the Memorystore Redis instance

Congratulations, you have now started monitoring metrics from key Google Cloud services for your application!

What to monitor on Google Cloud next?

Add logs from Google Cloud Services

Now that metrics are being monitored, you can also now add logging. There are several options for ingesting logs.

The Google Cloud Platform Integration in the Elastic Agent has four separate logs settings: audit logs, firewall logs, VPC Flow logs, and DNS logs. Just ensure you turn on what you wish to receive.

Analyze your data with Elastic machine learning

Once metrics and logs (or either one) are in Elastic, start analyzing your data through Elastic’s ML capabilities. A great review of these features can be found here:

• Correlating APM Telemetry to determine root causes in transactions
• Introduction to Elastic Machine Learning

Conclusion: Monitoring Google Cloud service metrics with Elastic Observability is easy!

I hope you’ve gotten an appreciation for how Elastic Observability can help you monitor Google Cloud service metrics. Here’s a quick recap of lessons and what you learned:

• Elastic Observability supports ingest and analysis of Google Cloud service metrics.
• It’s easy to set up ingest from Google Cloud services via the Elastic Agent.
• Elastic Observability has multiple out-of-the-box Google Cloud service dashboards you can use to preliminarily review information and then modify for your needs.
• For metrics not covered by out-of-the-box dashboards, custom dashboards can be easily created to visualize metrics that are important to you.
• 16 Google Cloud services are supported as part of Google Cloud Platform Integration on Elastic Observability, with more services being added regularly.
• As noted in related blogs, you can analyze your Google Cloud service metrics with Elastic’s machine learning capabilities.

Try it out for yourself by signing up via Google Cloud Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on Google Cloud around the world. Your Google Cloud Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with Google Cloud.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic is committed to supporting Google Kubernetes Engine (GKE) in all of its delivery modes. In October, during the Google Cloud Next ‘22 event, we announced our intention to integrate and certify Elastic Agent on Anthos, Autopilot, Google Distributed Cloud, and more.

Since that event, we have worked together with Google to get the Elastic Agent certified for use on Anthos, but we didn’t stop there.

Today we are happy to announce that we have been certified for operation on GKE Autopilot.

Hands on with Elastic and GKE Autopilot

Kubernetes observability has never been easier

To show how easy it is to get started with Autopilot and Elastic, let's walk through deploying the Elastic Agent on an Autopilot cluster. I’ll show how easy it is to set up and monitor an Autopilot cluster with the Elastic Agent and observe the cluster’s behavior with Kibana integrations.

One of the main differences between GKE and GKE Autopilot is that Autopilot protects the system namespace “kube-system.” To increase the stability and security of a cluster, Autopilot prevents user space workloads from adding or modifying system pods. The default configuration for Elastic Agent is to install itself into the system namespace. The majority of the changes we will make here are to convince the Elastic Agent to run in a different namespace.

Let’s get started with Elastic Stack!

While writing this article, I used the latest version of Elastic. The best way for you to get started with Elastic Observability is to:

1. Get an account on Elastic Cloud and look at this tutorial to help launch your first stack, or
2. Launch Elastic Cloud on your Google Account

Provisioning an Autopilot cluster and an Elastic stack

To test the agent, I first deployed the recommended, default GKE Autopilot cluster. Elastic’s GKE integration supports kube-state-metrics (KSM), which will increase the number of reported metrics available for reporting and dashboards. Like the Elastic Agent, KSM defaults to running in the system namespace, so I modified its manifest to work with Autopilot. For my testing, I also deployed a basic Elastic stack on Elastic Cloud in the same Google region as my Autopilot cluster. I used a fresh cluster deployed on Elastic’s managed service (ESS), but the process is the same if you are using an Elastic Cloud subscription purchased through the Google marketplace.

Adding Elastic Observability to GKE Autopilot

Because this is a brand new deployment, Elastic suggests adding integrations to it. Let’s add the Kubernetes integration into the new deployment:

Elastic offers hundreds of integrations; filter the list by typing “kub” into the search bar (1) and then click the Kubernetes integration (2).

The Kubernetes integration page gives you an overview of the integration and lets you manage the Kubernetes clusters you want to observe. We haven’t added a cluster yet, so I clicked “Add Kubernetes” to add the first integration.

I changed the integration name to reflect the Kubernetes offering type and then clicked “Save and continue” to accept the integration defaults.

At this point, an Agent policy has been created. Now it’s time to install the agent. I clicked on the “Kubernetes” integration.

Then I selected the “integration policies” tab (1) and clicked “Add agent” (2).

Finally, I downloaded the full manifest for a standard GKE environment.

We won’t be using this manifest directly, but it contains many of the values that we will need to deploy the agent on Autopilot in the next section.

The Elastic stack is ready and waiting for the Autopilot logs, metrics, and events. It’s time to connect Autopilot to this deployment using the Elastic Agent for GKE.

Connect Autopilot to Elastic

From the Google cloud terminal, I downloaded and edited the Elastic Agent manifest for GKE Autopilot.

$ curl -o elastic-agent-managed-gke-autopilot.yaml \
https://github.com/elastic/elastic-agent/blob/autopilotdocumentaton/docs/manifests/elastic-agent-managed-gke-autopilot.yaml

I used the cloud shell editor to configure the manifest for my Autopilot and Elastic clusters. For example, I updated the following:

containers:
  - name: elastic-agent
    image: docker.elastic.co/beats/elastic-agent:8.19.13

I also changed the agent to the version of Elastic that I installed (8.6.0).

From the Integration manifest I downloaded earlier, I copied the values for FLEET_URL and FLEET_ENROLLMENT_TOKEN into this YAML file.

Now it’s time to apply the updated manifest to the Autopilot instance.

Before I commit, I always like to see what’s going to be created (and check for syntax errors) with a dry run.

$ clear
$ kubectl apply --dry-run="client" -f elastic-agent-managed-gke-autopilot.yaml

Everything looks good, so I’ll do it for real this time.

$ clear
$ kubectl apply -f elastic-agent-managed-gke-autopilot.yaml

After several minutes, metrics will start flowing from the Autopilot cluster directly into the Elastic deployment.

Adding a workload to the Autopilot cluster

Observing an Autopilot cluster without a workload is boring, so I deployed a modified version of Google’s Hipster Shop (which includes OpenTelemetry reporting):

$ git clone https://github.com/bshetti/opentelemetry-microservices-demo
$ cd opentelemetry-microservices-demo
$ nano ./deploy-with-collector-k8s/otelcollector.yaml

To get the application’s telemetry talking to our Elastic stack, I replaced all instances of the exporter type from HTTP (otlphttp/elastic) to gRPC (otlp/elastic). I then replaced OTEL_EXPORTER_OTLP_ENDPOINT with my APM endpoint and I replaced OTEL_EXPORTER_OTLP_HEADERS with my APM OTEL Bearer and Token.

Then I deployed the Hipster Shop.

$ kubectl create -f ./deploy-with-collector-k8s/adservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/redis.yaml
$ kubectl create -f ./deploy-with-collector-k8s/cartservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/checkoutservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/currencyservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/emailservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/frontend.yaml
$ kubectl create -f ./deploy-with-collector-k8s/paymentservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/productcatalogservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/recommendationservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/shippingservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/loadgenerator.yaml

Once all of the shop’s pods were running, I deployed the OpenTelemetry collector.

$ kubectl create -f ./deploy-with-collector-k8s/otelcollector.yaml

Observe and visualize Autopilot’s metrics

Now that we have added the Elastic Agent to our Autopilot cluster and added a workload, let's take a look at some of the Kubernetes visualizations the integration provides out of the box.

The “[Metrics Kubernetes] Overview” is a great place to start. It provides a high-level view of the resources used by the cluster and allows me to drill into more specific dashboards that I find interesting:

For example, the “[Metrics Kubernetes] Pods” gives me a high-level view of the pods deployed in the cluster:

The “[Metrics Kubernetes] Volumes” gives me an in-depth view to how storage is allocated and used in the Autopilot cluster:

Creating an alert

From here, I can easily discover patterns in my cluster’s behavior and even create Alerts. Here is an example of an alert to notify me if the the main storage volume (called “volume”) exceeds 80% of its allocated space:

With a little work, I created this view from the standard dashboard:

Conclusion

Today I have shown how easy it is to monitor, observe, and generate alerts on a GKE Autopilot cluster. To get more information on what is possible, see the official Elastic documentation for Autopilot observability with Elastic Agent.

Next steps

If you don’t have Elastic yet, you can get started for free with an Elastic Trial today. Get more from Elastic and Google together with a Marketplace subscription. Elastic does more than just integrate with GKE — check out the almost 300 integrations that Elastic provides.

New Elastic Observability LLM integration with Google Cloud’s Vertex AI platform

We are thrilled to announce general availability of monitoring LLMs hosted in Google Cloud through the Elastic integration with Vertex AI. This integration enables users to experience enhanced LLM Observability by providing deep insights into the usage, cost and operational performance of models on Vertex AI, including latency, errors, token usage, frequency of model invocations as well as resources utilized by models. By leveraging this data, organizations can optimize resource usage, identify and resolve performance bottlenecks, and enhance the model efficiency and accuracy.

Observability needs for AI-powered applications using the Vertex AI platform

Leveraging AI models creates unique needs around the observability and monitoring of AI-powered applications. Some of the challenges that come with using LLMs are related to the high cost to call the LLMs, the quality and safety of LLM responses, and the performance, reliability and availability of the LLMs.

Lack of visibility into LLM observability data can make it harder for SREs and DevOps teams to ensure their AI-powered applications meet their service level objectives for reliability, performance, cost and quality of the AI-generated content and have enough telemetry data to troubleshoot related issues. Thus, robust LLM observability and detection of anomalies in the performance of models hosted on Google Cloud’s Vertex AI platform in real time is critical for the success of AI-powered applications.

Depending on the needs of their LLM applications, customers can make use of a growing list of models hosted on the Vertex AI platform such as Gemini 2.0 Pro, Gemini 2.0 Flash, and Imagen for image generation. Each model excels in specific areas and generates content in some modalities including Language, Audio, Vision, Code, etc. No two models are the same; each model has specific performance characteristics. So, it is important that service operators are able to track the individual performance, behaviour and cost of each model.

Unlocking Insights with Vertex AI Metrics

The Elastic integration with Google Cloud’s Vertex AI platform collects a wide range of metrics from models hosted on Vertex AI, enabling users to monitor, analyze, and optimize their AI deployments effectively.

Once you use the integration, you can review all the metrics in the Vertex AI dashboard

These metrics can be categorized into the following groups:

1. Prediction Metrics

Prediction metrics provide critical insights into model usage, performance bottlenecks, and reliability. These metrics help ensure smooth operations, optimize response times, and maintain robust, accurate predictions.

• Prediction Count by Endpoint: Measures the total number of predictions across different endpoints.

• Prediction Latency: Provides insights into the time taken to generate predictions, allowing users to identify bottlenecks in performance.

• Prediction Errors: Monitors the count of failed predictions across endpoints.

2. Model Performance Metrics

Model performance metrics provide crucial insights into deployment efficiency, and responsiveness. These metrics help optimize model performance and ensure reliable operations.

• Model Usage: Tracks the usage distribution among different model deployments.

• Token Usage: Tracks the number of tokens consumed by each model deployment, which is critical for understanding model efficiency.

• Invocation Rates: Tracks the frequency of invocations made by each model deployment.

• Model Invocation Latency: Measures the time taken to invoke a model, helping in diagnosing performance issues.

3. Resource Utilization Metrics

Resource utilization metrics are vital for monitoring resource efficiency and workload performance. They help optimize infrastructure, prevent bottlenecks, and ensure smooth operation of AI deployments.

• CPU Utilization: Monitors CPU usage to ensure optimal resource allocation for AI workloads.

• Memory Usage: Tracks the memory consumed across all model deployments.

• Network Usage: Measures bytes sent and received, providing insights into data transfer during model interactions.

4. Overview Metrics

These metrics give an overview of the models deployed in Google Cloud’s Vertex AI platform. They are essential for tracking overall performance, optimizing efficiency, and identifying potential issues across deployments.

• Total Invocations: The overall count of prediction invocations across all models and endpoints, providing a comprehensive view of activity.

• Total Tokens: The total number of tokens processed across all model interactions, offering insights into resource utilization and efficiency.

• Total Errors: The total count of errors encountered across all models and endpoints, helping identify reliability issues.

All metrics can be filtered by region, offering localized insights for better analysis.

Note: The Elastic I integration with Vertex AI provides comprehensive visibility into both deployment models: provisioned throughput, where capacity is pre-allocated, and pay-as-you-go, where resources are consumed on demand.

Conclusion

This integration with Vertex AI represents a significant step forward in enhancing the LLM Observability for users of Google Cloud’s Vertex AI platform. By unlocking a wealth of actionable data, organizations can assess the health, performance and cost of LLMs and troubleshoot operational issues, ensuring scalability, and accuracy in AI-driven applications.

Now that you know how the Vertex AI integration enhances LLM Observability, it’s your turn to try it out n. Spin up an Elastic Cloud, and start monitoring your LLM applications hosted on Google Cloud’s Vertex AI platform.

This blog post will show you how to deploy a simple Hello World web app to Cloud Run and then walk you through the steps to instrument the Hello World web app to enable observation of the application’s operations with Elastic Cloud.

Elastic Observability setup

We’ll start with setting up an Elastic Cloud deployment, which is where observability will take place for the web app we’ll be deploying.

From the Elastic Cloud console, select Create deployment.

Enter a deployment name and click Create deployment. It takes a few minutes for your deployment to be created. While waiting, you are prompted to save the admin credentials for your deployment, which provides you with superuser access to your Elastic^® deployment. Keep these credentials safe as they are shown only once.

Elastic Observability requires an APM Server URL and an APM Secret token for an app to send observability data to Elastic Cloud. Once the deployment is created, we’ll copy the Elastic Observability server URL and secret token and store them somewhere safely for adding to our web app code in a later step.

To copy the APM Server URL and the APM Secret Token, go to Elastic Cloud. Then go to the Deployments page which lists all of the deployments you have created. Select the deployment you want to use, which will open the deployment details page. In the Kibana row of links, click on Open to open Kibana for your deployment.

Select Integrations from the top-level menu. Then click the APM tile.

On the APM Agents page, copy the secretToken and the serverUrl values and save them for use in a later step.

Now that we’ve completed the Elastic Cloud setup, the next step is to set up our Google Cloud project for deploying apps to Cloud Run.

Google Cloud Run setup

First we’ll need a Google Cloud project, so let’s create one by going to the Google Cloud console and creating a new project. Select the project menu and then click the New Project button.

Once the new project is created, we’ll need to enable the necessary APIs that our Hello World app will be using. This can be done by clicking this enable APIs link, which opens a page in the Google Cloud console that lists the APIs that will be enabled and allows us to confirm their activation.

After we’ve enabled the necessary APIs, we’ll need to set up the required permissions for our Hello World app, which can be done in the IAM section of the Google Cloud Console. Within the IAM section, select the Compute Engine default service account and add the following roles:

• Logs Viewer
• Monitoring Viewer
• Pub/Sub Subscriber

Deploy a Hello World web app to Cloud Run

We’ll perform the process of deploying a Node.js Hello World web app to Cloud Run using the handy Google Cloud tool called Cloud Shell Editor. To deploy the Hello World app, we’ll perform the following five steps:

1. In Cloud Shell Editor, in the terminal window that appears at the bottom of the screen, clone a Node.js Hello World sample app repo from GitHub by entering the following command.

git clone https://github.com/elastic/observability-examples

2. Change directory to the location of the Hello World web app code.

cd gcp/run/helloworld

3. Build the Hello World app image and push the image to Google Container Registry by running the command below in the terminal. Be sure to replace your-project-id in the command below with your actual Google Cloud project ID.

gcloud builds submit --tag gcr.io/your-project-id/elastic-helloworld

4. Deploy the Hello World app to Google Cloud Run by running the command below. Be sure to replace your-project-id in the command below with your actual Google Cloud project ID.

gcloud run deploy elastic-helloworld --image gcr.io/your-project-id/elastic-helloworld

5. When the deployment process is complete, a Service URL will be displayed within the terminal. Copy and paste the Service URL in a browser to view the Hello World app running in Cloud Run.

Instrument the Hello World web app with Elastic Observability

With a web app successfully running in Cloud Run, we’re now ready to add the minimal code necessary to start monitoring the app. To enable observability for the Hello World app in Elastic Cloud, we’ll perform the following six steps:

1. In the Google Cloud Shell Editor, edit the Dockerfile file to add the following Elastic Open Telemetry environment variables along with the commands to install and run the Elastic APM agent. Replace the ELASTIC_APM_SERVER_URL text and the ELASTIC_APM_SECRET_TOKEN text with the APM Server URL and the APM Secret Token values that you copied and saved in an earlier step.

ENV OTEL_EXPORTER_OTLP_ENDPOINT='ELASTIC_APM_SERVER_URL'
ENV OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer ELASTIC_APM_SECRET_TOKEN'
ENV OTEL_LOG_LEVEL=info
ENV OTEL_METRICS_EXPORTER=otlp
ENV OTEL_RESOURCE_ATTRIBUTES=service.version=1.0,deployment.environment=production
ENV OTEL_SERVICE_NAME=helloworld
ENV OTEL_TRACES_EXPORTER=otlp
RUN npm install --save @opentelemetry/api
RUN npm install --save @opentelemetry/auto-instrumentations-node
CMD ["node", "--require", "@opentelemetry/auto-instrumentations-node/register", "index.js"]

The updated Dockerfile should look something like this:

FROM node:18-slim
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --only=production
COPY . ./
OTEL_EXPORTER_OTLP_ENDPOINT='https://******.apm.us-central1.gcp.cloud.es.io:443'
ENV OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer ******************'
ENV OTEL_LOG_LEVEL=info
ENV OTEL_METRICS_EXPORTER=otlp
ENV OTEL_RESOURCE_ATTRIBUTES=service.version=1.0,deployment.environment=production
ENV OTEL_SERVICE_NAME=helloworld
ENV OTEL_TRACES_EXPORTER=otlp
RUN npm install --save @opentelemetry/api
RUN npm install --save @opentelemetry/auto-instrumentations-node
CMD ["node", "--require", "@opentelemetry/auto-instrumentations-node/register", "index.js"]

2. In the Google Cloud Shell Editor, edit the package.json file to add the Elastic APM dependency. The dependencies section in package.json should look something like this:

"dependencies": {
  	"express": "^4.18.2",
  	"elastic-apm-node": "^3.49.1"
  },

3. In the Google Cloud Shell Editor, edit the index.js file:

• Add the code required to initialize the Elastic Open Telemetry APM agent:

const otel = require("@opentelemetry/api");
const tracer = otel.trace.getTracer("hello-world");

• Replace the “Hello World!” output code . . .

res.send(`<h1>Hello World!</h1>`);

...with the “Hello Elastic Observability” code block.

res.send(
  `<div style="text-align: center;">
   <h1 style="color: #005A9E; font-family:'Verdana'">
   Hello Elastic Observability - Google Cloud Run - Node.js
   </h1>
   <img src="https://storage.googleapis.com/elastic-helloworld/elastic-logo.png">
   </div>`
);

• Add a trace “hi” before the “Hello Elastic Observability” code block and add a trace “bye” after the “Hello Elastic Observability” code block.

tracer.startActiveSpan("hi", (span) => {
  console.log("hello");
  span.end();
});
res.send(
  `<div style="text-align: center;">
   <h1 style="color: #005A9E; font-family:'Verdana'">
   Hello Elastic Observability - Google Cloud Run - Node.js
   </h1>
   <img src="https://storage.googleapis.com/elastic-helloworld/elastic-logo.png">
   </div>`
);
tracer.startActiveSpan("bye", (span) => {
  console.log("goodbye");
  span.end();
});

• The completed index.js file should look something like this:

const otel = require("@opentelemetry/api");
const tracer = otel.trace.getTracer("hello-world");

const express = require("express");
const app = express();

app.get("/", (req, res) => {
  tracer.startActiveSpan("hi", (span) => {
    console.log("hello");
    span.end();
  });
  res.send(
    `<div style="text-align: center;">
    <h1 style="color: #005A9E; font-family:'Verdana'">
    Hello Elastic Observability - Google Cloud Run - Node.js
    </h1>
   <img src="https://storage.googleapis.com/elastic-helloworld/elastic-logo.png">
    </div>`
  );
  tracer.startActiveSpan("bye", (span) => {
    console.log("goodbye");
    span.end();
  });
});

const port = parseInt(process.env.PORT) || 8080;
app.listen(port, () => {
  console.log(`helloworld: listening on port ${port}`);
});

4. Rebuild the Hello World app image and push the image to the Google Container Registry by running the command below in the terminal. Be sure to replace your-project-id in the command below with your actual Google Cloud project ID.

gcloud builds submit --tag gcr.io/your-project-id/elastic-helloworld

5. Redeploy the Hello World app to Google Cloud Run by running the command below. Be sure to replace your-project-id in the command below with your actual Google Cloud project ID.

gcloud run deploy elastic-helloworld --image gcr.io/your-project-id/elastic-helloworld

6. When the deployment process is complete, a Service URL will be displayed within the terminal. Copy and paste the Service URL in a browser to view the updated Hello World app running in Cloud Run.

Observe the Hello World web app

Now that we’ve instrumented the web app to send observability data to Elastic Observability, we can now use Elastic Cloud to monitor the web app’s operations.

1. In Elastic Cloud, select the Observability Services menu item.

2. Click the helloworld service.

3. Click the Transactions tab.

4. Scroll down and click the GET / transaction.

5. Scroll down to the Trace Sample section to see the GET / , hi and bye trace samples.

Observability made to scale

You’ve seen the entire process of deploying a web app to Google Cloud Run that is instrumented with Elastic Observability. The end result is a web app that will scale up and down with demand combined with the observability tools to monitor the web app as it serves a single user or millions of users.

Now that you’ve seen how to deploy a serverless web app instrumented with observability, visit Elastic Observability to learn more about how to implement a complete observability solution for your apps. Or visit Getting started with Elastic on Google Cloud for more examples of how you can drive the data insights you need by combining Google Cloud monitoring and cloud computing services with Elastic’s search-powered platform.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The day-to-day operations of understanding the status and health of your Kubernetes clusters and applications running on them, through the logs, metrics, and traces they generate, will likely be your biggest challenge. But as an operations engineer you will need all of that important data to help prevent, predict, and remediate issues. And you certainly don’t need that volume of metrics, logs and traces spread across multiple tools when you need to visualize and analyze Kubernetes telemetry data for troubleshooting and support.

Elastic Observability helps manage the sprawl of Kubernetes metrics and logs by providing extensive and centralized observability capabilities beyond just the logging that we are known for. Elastic Observability provides you with granular insights and context into the behavior of your Kubernetes clusters along with the applications running on them by unifying all of your metrics, log, and trace data through OpenTelemetry and APM agents.

Regardless of the cluster location (EKS, GKE, AKS, self-managed) or application, Kubernetes monitoring is made simple with Elastic Observability. All of the node, pod, container, application, and infrastructure (AWS, GCP, Azure) metrics, infrastructure and application logs, along with application traces are available in Elastic Observability.

In this blog we will show:

• How Elastic Cloud can aggregate and ingest metrics and log data through the Elastic Agent (easily deployed on your cluster as a DaemonSet) to retrieve logs and metrics from the host (system metrics, container stats) along with logs from all services running on top of Kubernetes.
• How Elastic Observability can bring a unified telemetry experience (logs, metrics,traces) across all your Kubernetes cluster components (pods, nodes, services, namespaces, and more).

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• While we used GKE, you can use any location for your Kubernetes cluster.
• We used a variant of the ever so popular HipsterShop demo application. It was originally written by Google to showcase Kubernetes across a multitude of variants available such as the OpenTelemetry Demo App. To use the app, please go here and follow the instructions to deploy. You don’t need to deploy otelcollector for Kubernetes metrics to flow — we will cover this below.
• Elastic supports native ingest from Prometheus and FluentD, but in this blog, we are showing a direct ingest from Kubernetes cluster via Elastic Agent. There will be a follow-up blog showing how Elastic can also pull in telemetry from Prometheus or FluentD/bit.

What can you observe and analyze with Elastic?

Before we walk through the steps on getting Elastic set up to ingest and visualize Kubernetes cluster metrics and logs, let’s take a sneak peek at Elastic’s helpful dashboards.

As we noted, we ran a variant of HipsterShop on GKE and deployed Elastic Agents with Kubernetes integration as a DaemonSet on the GKE cluster. Upon deployment of the agents, Elastic starts ingesting metrics from the Kubernetes cluster (specifically from kube-state-metrics) and additionally Elastic will pull all log information from the cluster.

Visualizing Kubernetes metrics on Elastic Observability

Here are a few Kubernetes dashboards that will be available out of the box (OOTB) on Elastic Observability.

In addition to the cluster overview dashboard and pod dashboard, Elastic has several useful OOTB dashboards:

• Kubernetes overview dashboard (see above)
• Kubernetes pod dashboard (see above)
• Kubernetes nodes dashboard
• Kubernetes deployments dashboard
• Kubernetes DaemonSets dashboard
• Kubernetes StatefulSets dashboards
• Kubernetes CronJob & Jobs dashboards
• Kubernetes services dashboards
• More being added regularly

Additionally, you can either customize these dashboards or build out your own.

Working with logs on Elastic Observability

As you can see from the screens above, not only can I get Kubernetes cluster metrics, but also all the Kubernetes logs simply by using the Elastic Agent in my Kubernetes cluster.

Prevent, predict, and remediate issues

In addition to helping manage metrics and logs, Elastic can help you detect and predict anomalies across your cluster telemetry. Simply turn on Machine Learning in Elastic against your data and watch it help you enhance your analysis work. As you can see below, Elastic is not only a unified observability location for your Kubernetes cluster logs and metrics, but it also provides extensive true machine learning capabilities to enhance your analysis and management.

In the top graph, you see anomaly detection across logs and it shows something potentially wrong in the September 21 to 23 time period. Dig into the details on the bottom chart by analyzing a single kubernetes.pod.cpu.usage.node metric showing cpu issues early in September and again, later on in the month. You can do more complicated analyses on your cluster telemetry with Machine Learning using multi-metric analysis (versus the single metric issue I am showing above) along with population analysis.

Elastic gives you better machine learning capabilities to enhance your analysis of Kubernetes cluster telemetry. In the next section, let’s walk through how easy it is to get your telemetry data into Elastic.

Setting it all up

Let’s walk through the details of how to get metrics, logs, and traces into Elastic from a HipsterShop application deployed on GKE.

First, pick your favorite version of Hipstershop — as we noted above, we used a variant of the OpenTelemetry-Demo because it already has OTel. We slimmed it down for this blog, however (fewer services with some varied languages).

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Get a Kubernetes cluster and load your Kubernetes app into your cluster

Get your app on a Kubernetes cluster in your Cloud service of choice or local Kubernetes platform. Once your app is up on Kubernetes, you should have the following pods (or some variant) running on the default namespace.

NAME                                    READY   STATUS    RESTARTS   AGE
adservice-8694798b7b-jbfxt              1/1     Running   0          4d3h
cartservice-67b598697c-hfsxv            1/1     Running   0          4d3h
checkoutservice-994ddc4c4-p9p2s         1/1     Running   0          4d3h
currencyservice-574f65d7f8-zc4bn        1/1     Running   0          4d3h
emailservice-6db78645b5-ppmdk           1/1     Running   0          4d3h
frontend-5778bfc56d-jjfxg               1/1     Running   0          4d3h
jaeger-686c775fbd-7d45d                 1/1     Running   0          4d3h
loadgenerator-c8f76d8db-gvrp7           1/1     Running   0          4d3h
otelcollector-5b87f4f484-4wbwn          1/1     Running   0          4d3h
paymentservice-6888bb469c-nblqj         1/1     Running   0          4d3h
productcatalogservice-66478c4b4-ff5qm   1/1     Running   0          4d3h
recommendationservice-648978746-8bzxc   1/1     Running   0          4d3h
redis-cart-96d48485f-gpgxd              1/1     Running   0          4d3h
shippingservice-67fddb767f-cq97d        1/1     Running   0          4d3h

Step 2: Turn on <a href="https://github.com/kubernetes/kube-state-metrics" target="_self">kube-state-metrics</a>

Next you will need to turn on kube-state-metrics.

First:

git clone https://github.com/kubernetes/kube-state-metrics.git

Next, in the kube-state-metrics directory under the examples directory, just apply the standard config.

kubectl apply -f ./standard

This will turn on kube-state-metrics, and you should see a pod similar to this running in kube-system namespace.

kube-state-metrics-5f9dc77c66-qjprz                    1/1     Running   0          4d4h

Step 3: Install the Elastic Agent with Kubernetes integration

Add Kubernetes Integration:

1. 
2. In Elastic, go to integrations and select the Kubernetes Integration, and select to Add Kubernetes.
3. Select a name for the Kubernetes integration.
4. Turn on kube-state-metrics in the configuration screen.
5. Give the configuration a name in the new-agent-policy-name text box.
6. Save the configuration. The integration with a policy is now created.

You can read up on the agent policies and how they are used on the Elastic Agent here.

1. Add Kubernetes integration.
2. Select the policy you just created in the second.
3. In the third step of Add Agent instructions, copy and paste or download the manifest.
4. Add manifest to the shell where you have kubectl running, save it as elastic-agent-managed-kubernetes.yaml, and run the following command.

kubectl apply -f elastic-agent-managed-kubernetes.yaml

You should see a number of agents come up as part of a DaemonSet in kube-system namespace.

NAME                                                   READY   STATUS    RESTARTS   AGE
elastic-agent-qr6hj                                    1/1     Running   0          4d7h
elastic-agent-sctmz                                    1/1     Running   0          4d7h
elastic-agent-x6zkw                                    1/1     Running   0          4d7h
elastic-agent-zc64h                                    1/1     Running   0          4d7h

In my cluster, I have four nodes and four elastic-agents started as part of the DaemonSet.

Step 4: Look at Elastic out of the box dashboards (OOTB) for Kubernetes metrics and start discovering Kubernetes logs

That is it. You should see metrics flowing into all the dashboards. To view logs for specific pods, simply go into Discover in Kibana and search for a specific pod name.

Additionally, you can browse all the pod logs directly in Elastic.

In the above example, I searched for frontendService and cartService logs.

Step 5: Bonus!

Because we were using an OTel based application, Elastic can even pull in the application traces. But that is a discussion for another blog.

Here is a quick peek at what Hipster Shop’s traces for a front end transaction look like in Elastic Observability.

Conclusion: Elastic Observability rocks for Kubernetes monitoring

I hope you’ve gotten an appreciation for how Elastic Observability can help you manage Kubernetes clusters along with the complexity of the metrics, log, and trace data it generates for even a simple deployment.

A quick recap of lessons and more specifically learned:

• How Elastic Cloud can aggregate and ingest telemetry data through the Elastic Agent, which is easily deployed on your cluster as a DaemonSet and retrieves metrics from the host, such as system metrics, container stats, and metrics from all services running on top of Kubernetes
• Show what Elastic brings from a unified telemetry experience (Kubernenetes logs, metrics, traces) across all your Kubernetes cluster components (pods, nodes, services, any namespace, and more).
• Interest in exploring Elastic’s ML capabilities which will reduce your MTTHH (mean time to happy hour)

Ready to get started? Register and try out the features and capabilities I’ve outlined above.

When using the OpenTelemetry Java SDK or the OpenTelemetry instrumentation agent, those attributes are not automatically captured by default. In this article we will show you first how to enable them with the SDK, then using the instrumentation agent and then we will show you how using the Elastic OpenTelemetry Distribution makes it even easier.

OpenTelemetry Java SDK

The OpenTelemetry Java SDK does not capture any cloud resource attributes, however it provides a pluggable service provider interface to register resource attributes providers and application developers have to provide the implementations.

Implementations for GCP and AWS are already included in the OpenTelemetry Java Contrib repo, so if you are using one of those cloud providers then it's mostly a matter of adding those providers to your application dependencies. Thanks to autoconfiguration those should be automatically included and enabled once they are added to the application classpath. The SDK documentation provides all the details to add and configure those in your application.

If you are using a cloud provider for which no such implementation is available, then you still have the option to provide your own which is a straightforward implementation of the ResourceProvider SPI (Service Provider Interface). In order to keep things consistent, you will have to rely on the existing cloud semantic conventions.

For example here is an example of a simple cloud resource attributes provider for a fictitious cloud provider named "potatoes".

package potatoes;

import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
import io.opentelemetry.sdk.autoconfigure.spi.ResourceProvider;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.semconv.incubating.CloudIncubatingAttributes;

public class PotatoesResourceProvider implements ResourceProvider {

@Override
public Resource createResource(ConfigProperties configProperties) {
   return Resource.create(Attributes.of(
           CloudIncubatingAttributes.CLOUD_PROVIDER, "potatoes",
           CloudIncubatingAttributes.CLOUD_PLATFORM, "french-fries",
           CloudIncubatingAttributes.CLOUD_REGION, "garden"
           ));
  }
}

OpenTelemetry Java instrumentation

The OpenTelemetry Java Instrumentation provides a java agent that instruments the application at runtime automatically for an extensive set of frameworks and libraries (see supported technologies).

Using instrumentation means that the application bytecode and the embedded libraries are modified automatically to make them behave as if explicit modifications were made in their source code to call the OpenTelemetry SDK in order to create traces, spans and metrics.

When an application is deployed with the OpenTelemetry instrumentation agent, the cloud resource attributes for GCP and AWS are included but not enabled by default since version 2.2.0. You can enable them through configuration by setting the following properties:

• For AWS: otel.resource.providers.aws.enabled=true

• For GCP: otel.resource.providers.gcp.enabled=true

Elastic OpenTelemetry Java Distribution

The Elastic OpenTelemetry Java distribution relies on the OpenTelemetry Java instrumentation which we often refer to as the Vanilla OpenTelemetry, and it thus inherits all of its features.

One major difference though is that the resource attributes providers for GCP and AWS are included and enabled by default to provide a better onboarding experience without extra configuration.

The minor cost to this is that it might make the application startup slightly slower due to having to call an HTTP(S) endpoint. This overhead is usually negligible compared to application startup but can become significant for some setups.

In order to reduce the startup overhead, or when the cloud provider is known in advance, you can selectively disable unused provider implementations through configuration:

• For AWS: otel.resource.providers.aws.enabled=false

• For GCP: otel.resource.providers.gcp.enabled=false

Conclusion

With this blogpost we have introduced what OpenTelemetry cloud resource attributes are and how they can be used and configured into application deployments using either OpenTelemetry SDK/API and Instrumentation agents.

When using the Elastic OpenTelemetry Java distribution, those resource providers are automatically provided and enabled for an easy and simple onboarding experience.

Another very interesting aspect of the cloud resource attribute providers available in the opentelemetry-java-contrib repository is that they are maintained by their respective vendors (Google and Amazon). For the end-user it means those implementations should be quite well tested and be robust to changes in the underlying infrastructure. For solution vendors like Elastic, it means we don't have to re-implement and reverse-engineer the infrastructure details of every cloud provider, hence proving that investing in those common components is a net win for the broader OpenTelemetry community.

In order to keep pace with demand, most of these online apps and services (for example, mobile applications, web pages, SaaS) are moving to a distributed microservice-based architecture and Kubernetes. Once you’ve migrated your app to the cloud, how do you manage and monitor production, scale, and availability of the service? OpenTelemetry is quickly becoming the de facto standard for instrumentation and collecting application telemetry data for Kubernetes applications.

OpenTelemetry (OTel) is an open source project providing a collection of tools, APIs, and SDKs that can be used to generate, collect, and export telemetry data (metrics, logs, and traces) to understand software performance and behavior. OpenTelemetry recently became a CNCF incubating project and has a significant amount of growing community and vendor support.

While OTel provides a standard way to instrument applications with a standard telemetry format, it doesn’t provide any backend or analytics components. Hence using OTel libraries in applications, infrastructure, and user experience monitoring provides flexibility in choosing the appropriate observability tool of choice. There is no longer any vendor lock-in for application performance monitoring (APM).

Elastic Observability natively supports OpenTelemetry and its OpenTelemetry protocol (OTLP) to ingest traces, metrics, and logs. All of Elastic Observability’s APM capabilities are available with OTel data. Hence the following capabilities (and more) are available for OTel data:

• Service maps
• Service details (latency, throughput, failed transactions)
• Dependencies between services
• Transactions (traces)
• ML correlations (specifically for latency)
• Service logs

In addition to Elastic’s APM and unified view of the telemetry data, you will now be able to use Elastic’s powerful machine learning capabilities to reduce the analysis, and alerting to help reduce MTTR.

Given its open source heritage, Elastic also supports other CNCF based projects, such as Prometheus, Fluentd, Fluent Bit, Istio, Kubernetes (K8S), and many more.

This blog will show:

• How to get a popular OTel instrumented demo app (Hipster Shop) configured to ingest into Elastic Cloud through a few easy steps
• Highlight some of the Elastic APM capabilities and features around OTel data and what you can do with this data once it’s in Elastic

In follow-up blogs, we will detail how to use Elastic’s machine learning with OTel telemetry data, how to instrument OTel application metrics for specific languages, how we can support Prometheus ingest through the OTel collector, and more. Stay tuned!

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• We used the OpenTelemetry Demo. Directions for using Elastic with OpenTelemetry Demo are here.
• Make sure you have kubectl and helm also installed locally.
• Additionally, we are using an OTel manually instrumented version of the application. No OTel automatic instrumentation was used in this blog configuration.
• Location of our clusters. While we used Google Kubernetes Engine (GKE), you can use any Kubernetes platform of your choice.
• While Elastic can ingest telemetry directly from OTel instrumented services, we will focus on the more traditional deployment, which uses the OpenTelemetry Collector.
• Prometheus and FluentD/Fluent Bit — traditionally used to pull all Kubernetes data — is not being used here versus Kubernetes Agents. Follow-up blogs will showcase this.

Here is the configuration we will get set up in this blog:

Setting it all up

Over the next few steps, I’ll walk through an Opentelemetry visualization:

• Getting an account on Elastic Cloud
• Bringing up a GKE cluster
• Bringing up the application
• Configuring Kubernetes OTel Collector configmap to point to Elastic Cloud
• Using Elastic Observability APM with OTel data for improved visibility

Step 0: Create an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Bring up a K8S cluster

We used Google Kubernetes Engine (GKE), but you can use any Kubernetes platform of your choice.

There are no special requirements for Elastic to collect OpenTelemetry data from a Kubernetes cluster. Any normal Kubernetes cluster on GKE, EKS, AKS, or Kubernetes compliant cluster (self-deployed and managed) works.

Step 2: Load the OpenTelemetry demo application on the cluster

Get your application on a Kubernetes cluster in your cloud service of choice or local Kubernetes platform. The application I am using is available here.

First clone the directory locally:

git clone https://github.com/elastic/opentelemetry-demo.git

(Make sure you have kubectl and helm also installed locally.)

The instructions utilize a specific opentelemetry-collector configuration for Elastic. Essentially, the Elastic values.yaml file specified in the elastic/opentelemetry-demo configure the opentelemetry-collector to point to the Elastic APM Server using two main values:

OTEL_EXPORTER_OTLP_ENDPOINT is Elastic’s APM Server
OTEL_EXPORTER_OTLP_HEADERS Elastic Authorization

These two values can be found in the OpenTelemetry setup instructions under the APM integration instructions (Integrations->APM) in your Elastic cloud.

Once you obtain this, the first step is to create a secret key on the cluster with your Elastic APM server endpoint, and your APM Secret Token with the following instruction:

kubectl create secret generic elastic-secret \
  --from-literal=elastic_apm_endpoint='YOUR_APM_ENDPOINT_WITHOUT_HTTPS_PREFIX' \
  --from-literal=elastic_apm_secret_token='YOUR_APM_SECRET_TOKEN'

Don't forget to replace:

• YOUR_APM_ENDPOINT_WITHOUT_HTTPS_PREFIX: your Elastic APM endpoint ( without https:// prefix ) with OTEL_EXPORTER_OTLP_ENDPOINT
• YOUR_APM_SECRET_TOKEN: your Elastic APM secret token OTEL_EXPORTER_OTLP_HEADERS

Now execute the following commands:

# switch to the kubernetes/elastic-helm directory
cd kubernetes/elastic-helm

# add the open-telemetry Helm repostiroy
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

# deploy the demo through helm install
helm install -f values.yaml my-otel-demo open-telemetry/opentelemetry-demo

Once your application is up on Kubernetes, you will have the following pods (or some variant) running on the default namespace.

kubectl get pods -n default

Output should be similar to the following:

NAME                                                  READY   STATUS    RESTARTS      AGE
my-otel-demo-accountingservice-5c77754b4f-vwph6       1/1     Running   0             5d4h
my-otel-demo-adservice-6b8b7c7dc5-mb7j5               1/1     Running   0             5d4h
my-otel-demo-cartservice-76d94b7dcd-2g4lf             1/1     Running   0             5d4h
my-otel-demo-checkoutservice-988bbdb88-hmkrp          1/1     Running   0             5d4h
my-otel-demo-currencyservice-6cf4b5f9f6-vz9t2         1/1     Running   0             5d4h
my-otel-demo-emailservice-868c98fd4b-lpr7n            1/1     Running   6 (18h ago)   5d4h
my-otel-demo-featureflagservice-8446ff9c94-lzd4w      1/1     Running   0             5d4h
my-otel-demo-ffspostgres-867945d9cf-zzwd7             1/1     Running   0             5d4h
my-otel-demo-frauddetectionservice-5c97c589b9-z8fhz   1/1     Running   0             5d4h
my-otel-demo-frontend-d85ccf677-zg9fp                 1/1     Running   0             5d4h
my-otel-demo-frontendproxy-6c5c4fccf6-qmldp           1/1     Running   0             5d4h
my-otel-demo-kafka-68bcc66794-dsbr6                   1/1     Running   0             5d4h
my-otel-demo-loadgenerator-64c545b974-xfccq           1/1     Running   1 (36h ago)   5d4h
my-otel-demo-otelcol-fdfd9c7cf-6lr2w                  1/1     Running   0             5d4h
my-otel-demo-paymentservice-7955c68859-ff7zg          1/1     Running   0             5d4h
my-otel-demo-productcatalogservice-67c879657b-wn2wj   1/1     Running   0             5d4h
my-otel-demo-quoteservice-748d754ffc-qcwm4            1/1     Running   0             5d4h
my-otel-demo-recommendationservice-df78894c7-lwm5v    1/1     Running   0             5d4h
my-otel-demo-redis-7d48567546-h4p4t                   1/1     Running   0             5d4h
my-otel-demo-shippingservice-f6fc76ddd-2v7qv          1/1     Running   0             5d4h

Step 3: Open Kibana and use the APM Service Map to view your OTel instrumented Services

In the Elastic Observability UI under APM, select servicemap to see your services.

If you are seeing this, then the OpenTelemetry Collector is sending data into Elastic:

Congratulations, you've instrumented the OpenTelemetry demo application using and successfully ingested the telemetry data into the Elastic!

Step 4: What can Elastic show me?

Now that the OpenTelemetry data is ingested into Elastic, what can you do?

First, you can view the APM service map (as shown in the previous step) — this will give you a full view of all the services and the transaction flows between services.

Next, you can now check out individual services and the transactions being collected.

As you can see, the frontend details are listed. Everything from:

• Average service latency
• Throughput
• Main transactions
• Failed traction rate
• Errors
• Dependencies

Let’s get to the trace. In the Transactions tab, you can review all the types of transactions related to the frontend service:

Selecting the HTTP POST transaction, we can see the full trace with all the spans:

Not only can you review the trace but you can also analyze what is related to higher than normal latency for HTTP POST .

Elastic uses machine learning to help identify any potential latency issues across the services from the trace. It’s as simple as selecting the Latency Correlations tab and running the correlation.

This shows that the high latency transactions are occurring in checkout service with a medium correlation.

You can then drill down into logs directly from the trace view and review the logs associated with the trace to help identify and pinpoint potential issues.

Analyze your data with Elastic machine learning (ML)

Once OpenTelemetry metrics are in Elastic, start analyzing your data through Elastic’s ML capabilities.

A great review of these features can be found here: Correlating APM telemetry to determine root causes in transactions. And there are many more videos and blogs on Elastic’s Blog. We’ll follow up with additional blogs on leveraging Elastic’s machine learning capabilities for OpenTelemetry data.

Conclusion

I hope you’ve gotten an appreciation for how Elastic Observability can help you ingest and analyze OpenTelemetry data with Elastic’s APM capabilities.

A quick recap of lessons and more specifically learned:

• How to get a popular OTel instrumented demo app (Hipster Shop) configured to ingest into Elastic Cloud, through a few easy steps
• Highlight some of the Elastic APM capabilities and features around OTel data and what you can do with this once it’s in Elastic

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your OpenTelemetry data.

First, we need an example application that allows us to monitor infrastructure changes effectively. We use a Python Flask application with the Elastic Python APM agent. The application is a simple calculator taking the numbers as a REST request. We utilize Locust — a simple load-testing tool to evaluate performance under varying workloads.

The next step includes obtaining the pricing information associated with the cloud services. Every cloud provider is different. Most of them offer an option to retrieve pricing through an API. But today, we will focus on Google Cloud and will leverage their pricing calculator to retrieve relevant cost information.

The calculator and Google Cloud pricing

To perform a cost analysis, we need to know the cost of the machines in use. Google provides a billing API and Client Library to fetch the necessary data programmatically. In this blog, we are not covering the API approach. Instead, the Google Cloud Pricing Calculator is enough. Select the machine type and region in the calculator and set the count 1 instance. It will then report the total estimated cost for this machine. Doing this for an e2-standard-4 machine type results in 107.7071784 US$ for a runtime of 730 hours.

Now, let’s go to our Kibana® where we will create a new index inside Dev Tools. Since we don’t want to analyze text, we will tell Elasticsearch® to treat every text as a keyword. The index name is cloud-billing. I might want to do the same for Azure and AWS, then I can append it to the same index.

PUT cloud-billing
{
  "mappings": {
    "dynamic_templates": [
      {
        "stringsaskeywords": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}

Next up is crafting our billing document:

POST cloud-billing/_doc/e2-standard-4_europe-west4
{
  "machine": {
    "enrichment": "e2-standard-4_europe-west4"
  },
  "cloud": {
    "machine": {
       "type": "e2-standard-4"
    },
    "region": "europe-west4",
    "provider": "google"
  },
  "stats": {
    "cpu": 4,
    "memory": 8
  },
  "price": {
    "minute": 0.002459068,
    "hour": 0.14754408,
    "month": 107.7071784
  }
}

We create a document and set a custom ID. This ID matches the instance name and the region since the machines' costs may differ in each region. Automatic IDs could be problematic because I might want to update what a machine costs regularly. I could use a timestamped index for that and only ever use the latest document matching. But this way, I can update and don’t have to worry about it. I calculated the price down to minute and hour prices as well. The most important thing is the machine.enrichment field, which is the same as the ID. The same instance type can exist in multiple regions, but our enrichment processor is limited to match or range. We create a matching name that can explicitly match as in e2-standard-4_europe-west4. It’s up to you to decide whether you want the cloud provider in there and make it google_e2-standard-4_europ-west-4.

Calculating the cost

There are multiple ways of achieving this in the Elastic Stack. In this case, we will use an enrich policy, ingest pipeline, and transform.

The enrich policy is rather easy to setup:

PUT _enrich/policy/cloud-billing
{
  "match": {
    "indices": "cloud-billing",
    "match_field": "machine.enrichment",
    "enrich_fields": ["price.minute", "price.hour", "price.month"]
  }
}

POST _enrich/policy/cloud-billing/_execute

Don’t forget to run the _execute at the end of it. This is necessary to make the internal indices used by the enrichment in the ingest pipeline. The ingest pipeline is rather minimalistic — it calls the enrichment and renames a field. This is where our machine.enrichment field comes in. One caveat around enrichment is that when you add new documents to the cloud-billing index, you need to rerun the _execute statement. The last bit calculates the total cost with the count of unique machines seen.

PUT _ingest/pipeline/cloud-billing
{
  "processors": [
    {
      "set": {
        "field": "_temp.machine_type",
        "value": "{{cloud.machine.type}}_{{cloud.region}}"
      }
    },
    {
      "enrich": {
        "policy_name": "cloud-billing",
        "field": "_temp.machine_type",
        "target_field": "enrichment"
      }
    },
    {
      "rename": {
        "field": "enrichment.price",
        "target_field": "price"
      }
    },
    {
      "remove": {
        "field": [
          "_temp",
          "enrichment"
        ]
      }
    },
    {
      "script": {
        "source": "ctx.total_price=ctx.count_machines*ctx.price.hour"
      }
    }
  ]
}

Since this is all configured now, we are ready for our Transform. For this, we need a data view that matches the APM data_streams. This is traces-apm*, metrics-apm.*, logs-apm.*. For the Transform, go to the Transform UI in Kibana and configure it in the following way:

We are doing an hourly breakdown, therefore, I get a document per service, per hour, per machine type. The interesting bit is the aggregations. I want to see the average CPU usage and the 75,95,99 percentile, to view the CPU usage on an hourly basis. Allowing me to identify the CPU usage across an hour. At the bottom, give the transform a name and select an index cloud-costs and select the cloud-billing ingest pipeline.

Here is the entire transform as a JSON document:

PUT _transform/cloud-billing
{
  "source": {
    "index": [
      "traces-apm*",
      "metrics-apm.*",
      "logs-apm.*"
    ],
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "exists": {
                    "field": "cloud.provider"
                  }
                }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1h"
        }
      },
      "cloud.provider": {
        "terms": {
          "field": "cloud.provider"
        }
      },
      "cloud.region": {
        "terms": {
          "field": "cloud.region"
        }
      },
      "cloud.machine.type": {
        "terms": {
          "field": "cloud.machine.type"
        }
      },
      "service.name": {
        "terms": {
          "field": "service.name"
        }
      }
    },
    "aggregations": {
      "avg_cpu": {
        "avg": {
          "field": "system.cpu.total.norm.pct"
        }
      },
      "percentiles_cpu": {
        "percentiles": {
          "field": "system.cpu.total.norm.pct",
          "percents": [
            75,
            95,
            99
          ]
        }
      },
      "avg_transaction_duration": {
        "avg": {
          "field": "transaction.duration.us"
        }
      },
      "percentiles_transaction_duration": {
        "percentiles": {
          "field": "transaction.duration.us",
          "percents": [
            75,
            95,
            99
          ]
        }
      },
      "count_machines": {
        "cardinality": {
          "field": "cloud.instance.id"
        }
      }
    }
  },
  "dest": {
    "index": "cloud-costs",
    "pipeline": "cloud-costs"
  },
  "sync": {
    "time": {
      "delay": "120s",
      "field": "@timestamp"
    }
  },
  "settings": {
    "max_page_search_size": 1000
  }
}

Once the transform is created and running, we need a Kibana Data View for the index: cloud-costs. For the transaction, use the custom formatter inside Kibana and set its format to “Duration” in “microseconds.”

With that, everything is arranged and ready to go.

Observing infrastructure changes

Below I created a dashboard that allows us to identify:

• How much costs a certain service creates
• CPU usage
• Memory usage
• Transaction duration
• Identify cost-saving potential

From left to right, we want to focus on the very first chart. We have the bars representing the CPU as average in green and 95th percentile in blue on top. It goes from 0 to 100% and is normalized, meaning that even with 8 CPU cores, it will still read 100% usage and not 800%. The line graph represents the transaction duration, the average being in red, and the 95th percentile in purple. Last, we have the orange area at the bottom, which is the average memory usage on that host.

We immediately realize that our calculator does not need a lot of memory. Hovering over the graph reveals 2.89% memory usage. The e2-standard-8 machine that we are using has 32 GB of memory. We occasionally spike to 100% CPU in the 95th percentile. When this happens, we see that the average transaction duration spikes to 2.5 milliseconds. However, every hour this machine costs us a rounded 30 cents. Using this information, we can now downsize to a better fit. The average CPU usage is around 11-13%, and the 95th percentile is not that far away.

Because we are using 8 CPUs, one could now say that 12.5% represents a full core, but that is just an assumption on a piece of paper. Nonetheless, we know there is a lot of headroom, and we can downscale quite a bit. In this case, I decided to go to 2 CPUs and 2 GB of RAM, known as e2-highcpu2. This should fit my calculator application better. We barely touched the RAM, 2.89% out of 32GB are roughly 1GB of use. After the change and reboot of the calculator machine, I started the same Locust test to identify my CPU usage and, more importantly, if my transactions get slower, and if so, by how much. Ultimately, I want to decide whether 1 millisecond more latency is worth 10 more cents per hour. I added the change as an annotation in Lens.

After letting it run for a bit, we can now identify the smaller hosts' impact. In this case, we can see that the average did not change. However, the 95th percentile — as in 95% of all transactions are below this value — did spike up. Again, it looks bad at first, but checking in, it went from ~1.5 milliseconds to ~2.10 milliseconds, a ~0.6 millisecond increase. Now, you can decide whether that 0.6 millisecond increase is worth paying ~180$ more per month or if the current latency is good enough.

Conclusion

Observability is more than just collecting logs, metrics, and traces. Linking user experience to cloud costs allows your business to identify areas where you can save money. Having the right tools at your disposal will help you generate those insights quickly. Making informed decisions about how to optimize your cloud cost and ultimately improve the user experience is the bottom-line goal.

The dashboard and data view can be found in my GitHub repository. You can download the .ndjson file and import it using the Saved Objects inside Stack Management in Kibana.

Caveats

Pricing is only for base machines without any disk information, static public IP addresses, and any other additional cost, such as licenses for operating systems. Furthermore, it excludes spot pricing, discounts, or free credits. Additionally, data transfer costs between services are also not included. We only calculate it based on the minute rate of the service running — we are not checking billing intervals from Google Cloud. In our case, we would bill per minute, regardless of what Google Cloud has. Using the count for unique instance.ids work as intended. However, if a machine is only running for one minute, we calculate it based on the hourly rate. So, a machine running for one minute, will cost the same as running for 50 minutes — at least how we calculate it. The transform uses calendar hour intervals; therefore, it's 8 am-9 am, 9 am-10 am, and so on.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.