Google Cloud Platform Integration

edit

Google Cloud Platform Integration

edit

Version

2.38.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Elastic

The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring.

Authentication

edit

To use this Google Cloud Platform (GCP) integration, you need to set up a Service Account with a Role and a Service Account Key to access data on your GCP project.

Service Account
edit

First, you need to create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources.

The Elastic Agent uses the SA to access data on Google Cloud Platform using the Google APIs.

If you haven’t already, this might be a good moment to check out the best practices for securing service accounts guide.

Role
edit

You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by Agent. Those privileges are:

  • compute.instances.list (required for GCP Compute instance metadata collection) **
  • monitoring.metricDescriptors.list
  • monitoring.timeSeries.list
  • pubsub.subscriptions.consume
  • pubsub.subscriptions.create *
  • pubsub.subscriptions.get
  • pubsub.topics.attachSubscription *
  • Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself you may omit these privileges.

    • Only required if corresponding collection will be enabled.

After you have created the custom role, assign the role to your service account.

Service Account Keys
edit

Now, with your brand new Service Account (SA) with access to Google Cloud Platform (GCP) resources, you need some credentials to associate with it: a Service Account Key.

From the list of SA:

  1. Click the one you just created to open the detailed view.
  2. From the Keys section, click "Add key" > "Create new key" and select JSON as the type.
  3. Download and store the generated private key securely (remember, the private key can’t be recovered from GCP if lost).

Configure the Integration Settings

edit

The next step is to configure the general integration settings used for all logs from the supported services (Audit, DNS, Firewall, and VPC Flow).

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.

Project Id
edit

The Project Id is the Google Cloud project ID where your resources exist.

Credentials File vs Json
edit

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.

Option 1: Credentials File
edit

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.

Option 2: Credentials JSON
edit

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.

Recommendations
edit

Elastic recommends using Credentials File, as in this method the credential information doesn’t leave your Google Cloud Platform environment. When using Credentials JSON, the integration stores the info in Elasticsearch, and the access is controlled based on policy permissions or access to underlying Elasticsearch data.

Logs Collection Configuration

edit

With a properly configured Service Account and the integration setting in place, it’s time to start collecting some logs.

Requirements
edit

You need to create a few dedicated Google Cloud resources before starting, in detail:

  • Log Sink
  • Pub/Sub Topic
  • Subscription

Elastic recommends separate Pub/Sub topics for each of the log types so that they can be parsed and stored in a specific data stream.

Here’s an example of collecting Audit Logs using a Pub/Sub topic, a subscription, and a Log Router. We will create the resources in the Google Cloud Console and then configure the Google Cloud Platform integration.

On the Google Cloud Console
edit

At a high level, the steps required are:

  • Visit "Logging" > "Log Router" > "Create Sink" and provide a sink name and description.
  • In "Sink destination", select "Cloud Pub/Sub topic" as the sink service. Select an existing topic or "Create a topic". Note the topic name, as it will be provided in the Topic field in the Elastic agent configuration.
  • If you created a new topic, you must remember to go to that topic and create a subscription for it. A subscription directs messages on a topic to subscribers. Note the "Subscription ID", as it will need to be entered in the "Subscription name" field in the integration settings.
  • Under "Choose logs to include in sink", for example add logName:"cloudaudit.googleapis.com" in the "Inclusion filter" to include all audit logs.

This is just an example; you will need to create your filter expression to select the log types you want to export to the Pub/Sub topic.

More example filters for different log types:

#
# VPC Flow: logs for specific subnet
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/vpc_flows") AND
resource.labels.subnetwork_name"=[SUBNET_NAME]"
#
# Audit: Google Compute Engine firewall rule deletion
#
resource.type="gce_firewall_rule" AND
log_id("cloudaudit.googleapis.com/activity") AND
protoPayload.methodName:"firewalls.delete"
#
# DNS: all DNS queries
#
resource.type="dns_query"
#
# Firewall: logs for a given country
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/firewall") AND
jsonPayload.remote_location.country=[COUNTRY_ISO_ALPHA_3]

Start working on your query using the Google Cloud Logs Explorer, so you can preview and pinpoint the exact log types you want to forward to your Elastic Stack.

To learn more, please read how to Build queries in the Logs Explorer, and take a look at the Sample queries using the Logs Explorer page in the Google Cloud docs.

On Kibana
edit

Visit "Management" > "Integrations" > "Installed Integrations" > "Google Cloud Platform" and select the "Integration Policies" tab. Select the integration policy you previously created.

From the list of services, select "Google Cloud Platform (GCP) audit logs (gcp-pubsub)" and:

  • On the "Topic" field, specify the "topic name" you noted before on the Google Cloud Console.
  • On the "Subscription Name", specify the short subscription name you noted before on the Google Cloud Console (note: do NOT use the full-blown subscription name made of project/PROJECT_ID/subscriptions/SUBSCRIPTION_ID). Just pick the Subscription ID from the Google Cloud Console).
  • Click on "Save Integration", and make sure the Elastic Agent gets the updated policy.
Troubleshooting
edit

If you don’t see Audit logs showing up, check the Agent logs to see if there are errors.

Common error types:

  • Missing roles in the Service Account
  • Misconfigured settings, like "Project Id", "Topic" or "Subscription Name" fields
Missing Roles in the Service Account
edit

If your Service Account (SA) does not have the required roles, you might find errors like this one in the elastic_agent.filebeat dataset:

failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = PermissionDenied desc = User not authorized to perform this action.

Solution: make sure your SA has all the required roles.

Misconfigured Settings
edit

If you specify the wrong "Topic field" or "Subscription Name", you might find errors like this one in the elastic_agent.filebeat dataset:

[elastic_agent.filebeat][error] failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = InvalidArgument desc = Invalid resource name given (name=projects/project/subscriptions/projects/project/subscriptions/non-existent-sub). Refer to https://cloud.google.com/pubsub/docs/admin#resource_names for more information.

Solution: double check the integration settings.

Metrics Collection Configuration

edit

With a properly configured Service Account and the integration setting in place, it’s time to start collecting some metrics.

Requirements
edit

No additional requirement is needed to collect metrics.

Troubleshooting
edit

If you don’t see metrics showing up, check the Agent logs to see if there are errors.

Common error types:

  • Period is lower than 60 seconds
  • Missing roles in the Service Account
  • Misconfigured settings, like "Project Id"
Period is lower than 60 seconds
edit

Usual minimum collection period for GCP metrics is 60 seconds. Any value lower than that cause an error when retrieving the metric metadata. If an error happens, the affected metric is skipped at the metric collection stage, resulting in no data being sent.

Missing Roles in the Service Account
edit

If your Service Account (SA) does not have required roles, you might find errors related to accessing GCP resources.

To check you may add Monitoring Viewer and Compute Viewer roles (built-in GCP roles) to your SA. These roles contain the permission added in the previous step and expand them with additional permissions. You can analyze additional missing permissions from the GCP Console > IAM > clicking on the down arrow near the roles on the same line of your SA > View analyzed permissions. From the shown table you can check which permissions from the role the SA is actively using. They should match what you configured in your custom role.

Misconfigured Settings
edit

If you specify a wrong setting you will probably find errors related to missing GCP resources.

Make sure the settings are correct and the SA has proper permissions for the given "Project Id".

Logs

edit
Audit
edit

The audit dataset collects audit logs of administrative activities and accesses within your Google Cloud resources.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.audit.authentication_info.authority_selector

The authority selector specified by the requestor, if any. It is not guaranteed that the principal was allowed to use this authority.

keyword

gcp.audit.authentication_info.principal_email

The email address of the authenticated user making the request.

keyword

gcp.audit.authentication_info.principal_subject

String representation of identity of requesting party. Populated for both first and third party identities. Only present for APIs that support third-party identities.

keyword

gcp.audit.authentication_info.service_account_delegation_info

Identity delegation history of an authenticated service account that makes the request. It contains information on the real authorities that try to access GCP resources by delegating on a service account. When multiple authorities present, they are guaranteed to be sorted based on the original ordering of the identity delegation events.

flattened

gcp.audit.authentication_info.service_account_key_name

The service account key that was used to request the OAuth 2.0 access token. This field identifies the service account key by its full resource name.

keyword

gcp.audit.authentication_info.third_party_principal

The third party identification (if any) of the authenticated user making the request. When the JSON object represented here has a proto equivalent, the proto name will be indicated in the @type property.

flattened

gcp.audit.authorization_info

Authorization information for the operation.

nested

gcp.audit.authorization_info.granted

Whether or not authorization for resource and permission was granted.

boolean

gcp.audit.authorization_info.permission

The required IAM permission.

keyword

gcp.audit.authorization_info.resource

The resource being accessed, as a REST-style string.

keyword

gcp.audit.authorization_info.resource_attributes.name

The name of the resource.

keyword

gcp.audit.authorization_info.resource_attributes.service

The name of the service.

keyword

gcp.audit.authorization_info.resource_attributes.type

The type of the resource.

keyword

gcp.audit.flattened

Contains the full audit document as sent by GCP.

flattened

gcp.audit.labels

A map of key, value pairs that provides additional information about the log entry. The labels can be user-defined or system-defined.

flattened

gcp.audit.logentry_operation.first

Optional. Set this to True if this is the first log entry in the operation.

boolean

gcp.audit.logentry_operation.id

Optional. An arbitrary operation identifier. Log entries with the same identifier are assumed to be part of the same operation.

keyword

gcp.audit.logentry_operation.last

Optional. Set this to True if this is the last log entry in the operation.

boolean

gcp.audit.logentry_operation.producer

Optional. An arbitrary producer identifier. The combination of id and producer must be globally unique.

keyword

gcp.audit.metadata

Service-specific data about the request, response, and other information associated with the current audited event.

flattened

gcp.audit.method_name

The name of the service method or operation. For API calls, this should be the name of the API method. For example, google.datastore.v1.Datastore.RunQuery.

keyword

gcp.audit.num_response_items

The number of items returned from a List or Query API method, if applicable.

long

gcp.audit.policy_violation_info.payload

Resource payload that is currently in scope and is subjected to orgpolicy conditions.

flattened

gcp.audit.policy_violation_info.resource_tags

Tags referenced on the resource at the time of evaluation.

flattened

gcp.audit.policy_violation_info.resource_type

Resource type that the orgpolicy is checked against.

keyword

gcp.audit.policy_violation_info.violations.checkedValue

Value that is being checked for the policy.

keyword

gcp.audit.policy_violation_info.violations.constraint

Constraint name.

keyword

gcp.audit.policy_violation_info.violations.errorMessage

Error message that policy is indicating.

keyword

gcp.audit.policy_violation_info.violations.policyType

Indicates the type of the policy.

keyword

gcp.audit.request

flattened

gcp.audit.request_metadata.caller_ip

The IP address of the caller.

ip

gcp.audit.request_metadata.caller_supplied_user_agent

The user agent of the caller. This information is not authenticated and should be treated accordingly.

keyword

gcp.audit.request_metadata.raw.caller_ip

The raw IP address of the caller.

keyword

gcp.audit.resource_location.current_locations

Current locations of the resource.

keyword

gcp.audit.resource_name

The resource or collection that is the target of the operation. The name is a scheme-less URI, not including the API service name. For example, shelves/SHELF_ID/books.

keyword

gcp.audit.response

flattened

gcp.audit.service_name

The name of the API service performing the operation. For example, datastore.googleapis.com.

keyword

gcp.audit.status.code

The status code, which should be an enum value of google.rpc.Code.

integer

gcp.audit.status.details

A list of messages that carry the error details.

flattened

gcp.audit.status.message

A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.

keyword

gcp.audit.type

Type property.

keyword

gcp.destination.instance.project_id

ID of the project containing the VM.

keyword

gcp.destination.instance.region

Region of the VM.

keyword

gcp.destination.instance.zone

Zone of the VM.

keyword

gcp.destination.vpc.project_id

ID of the project containing the VM.

keyword

gcp.destination.vpc.subnetwork_name

Subnetwork on which the VM is operating.

keyword

gcp.destination.vpc.vpc_name

VPC on which the VM is operating.

keyword

gcp.source.instance.project_id

ID of the project containing the VM.

keyword

gcp.source.instance.region

Region of the VM.

keyword

gcp.source.instance.zone

Zone of the VM.

keyword

gcp.source.vpc.project_id

ID of the project containing the VM.

keyword

gcp.source.vpc.subnetwork_name

Subnetwork on which the VM is operating.

keyword

gcp.source.vpc.vpc_name

VPC on which the VM is operating.

keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

Example

An example event for audit looks as following:

{
    "@timestamp": "2019-12-19T00:44:25.051Z",
    "agent": {
        "ephemeral_id": "a22278bb-5e1f-4ab7-b468-277c8c0b80a9",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "client": {
        "user": {
            "email": "xxx@xxx.xxx"
        }
    },
    "cloud": {
        "project": {
            "id": "elastic-beats"
        },
        "provider": "gcp"
    },
    "data_stream": {
        "dataset": "gcp.audit",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "beta.compute.instances.aggregatedList",
        "agent_id_status": "verified",
        "category": [
            "network",
            "configuration"
        ],
        "created": "2023-10-25T04:18:46.637Z",
        "dataset": "gcp.audit",
        "id": "yonau2dg2zi",
        "ingested": "2023-10-25T04:18:47Z",
        "kind": "event",
        "outcome": "success",
        "provider": "data_access",
        "type": [
            "access",
            "allowed"
        ]
    },
    "gcp": {
        "audit": {
            "authorization_info": [
                {
                    "granted": true,
                    "permission": "compute.instances.list",
                    "resource_attributes": {
                        "name": "projects/elastic-beats",
                        "service": "resourcemanager",
                        "type": "resourcemanager.projects"
                    }
                }
            ],
            "num_response_items": 61,
            "request": {
                "@type": "type.googleapis.com/compute.instances.aggregatedList"
            },
            "resource_location": {
                "current_locations": [
                    "global"
                ]
            },
            "resource_name": "projects/elastic-beats/global/instances",
            "response": {
                "@type": "core.k8s.io/v1.Status",
                "apiVersion": "v1",
                "details": {
                    "group": "batch",
                    "kind": "jobs",
                    "name": "gsuite-exporter-1589294700",
                    "uid": "2beff34a-945f-11ea-bacf-42010a80007f"
                },
                "kind": "Status",
                "status_value": "Success"
            },
            "type": "type.googleapis.com/google.cloud.audit.AuditLog"
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/elastic-beats/logs/cloudaudit.googleapis.com%2Fdata_access"
    },
    "service": {
        "name": "compute.googleapis.com"
    },
    "source": {
        "ip": "192.168.1.1"
    },
    "tags": [
        "forwarded",
        "gcp-audit"
    ],
    "user_agent": {
        "device": {
            "name": "Mac"
        },
        "name": "Firefox",
        "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0,gzip(gfe),gzip(gfe)",
        "os": {
            "full": "Mac OS X 10.15",
            "name": "Mac OS X",
            "version": "10.15"
        },
        "version": "71.0."
    }
}
Firewall
edit

The firewall dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.destination.instance.project_id

ID of the project containing the VM.

keyword

gcp.destination.instance.region

Region of the VM.

keyword

gcp.destination.instance.zone

Zone of the VM.

keyword

gcp.destination.vpc.project_id

ID of the project containing the VM.

keyword

gcp.destination.vpc.subnetwork_name

Subnetwork on which the VM is operating.

keyword

gcp.destination.vpc.vpc_name

VPC on which the VM is operating.

keyword

gcp.firewall.flattened

Contains the full firewall document as sent by GCP.

flattened

gcp.firewall.rule_details.action

Action that the rule performs on match.

keyword

gcp.firewall.rule_details.destination_range

List of destination ranges that the firewall applies to.

keyword

gcp.firewall.rule_details.direction

Direction of traffic that matches this rule.

keyword

gcp.firewall.rule_details.ip_port_info

List of ip protocols and applicable port ranges for rules.

nested

gcp.firewall.rule_details.priority

The priority for the firewall rule.

long

gcp.firewall.rule_details.reference

Reference to the firewall rule.

keyword

gcp.firewall.rule_details.source_range

List of source ranges that the firewall rule applies to.

keyword

gcp.firewall.rule_details.source_service_account

List of all the source service accounts that the firewall rule applies to.

keyword

gcp.firewall.rule_details.source_tag

List of all the source tags that the firewall rule applies to.

keyword

gcp.firewall.rule_details.target_service_account

List of all the target service accounts that the firewall rule applies to.

keyword

gcp.firewall.rule_details.target_tag

List of all the target tags that the firewall rule applies to.

keyword

gcp.source.instance.project_id

ID of the project containing the VM.

keyword

gcp.source.instance.region

Region of the VM.

keyword

gcp.source.instance.zone

Zone of the VM.

keyword

gcp.source.vpc.project_id

ID of the project containing the VM.

keyword

gcp.source.vpc.subnetwork_name

Subnetwork on which the VM is operating.

keyword

gcp.source.vpc.vpc_name

VPC on which the VM is operating.

keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

Example

An example event for firewall looks as following:

{
    "@timestamp": "2019-10-30T13:52:42.191Z",
    "agent": {
        "ephemeral_id": "175ae0b3-355c-4ca7-87ea-d5f1ee34102e",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "availability_zone": "us-east1-b",
        "project": {
            "id": "test-beats"
        },
        "provider": "gcp",
        "region": "us-east1"
    },
    "data_stream": {
        "dataset": "gcp.firewall",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "10.42.0.2",
        "domain": "test-windows",
        "ip": "10.42.0.2",
        "port": 3389
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "firewall-rule",
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "created": "2023-10-25T04:20:37.182Z",
        "dataset": "gcp.firewall",
        "id": "1f21ciqfpfssuo",
        "ingested": "2023-10-25T04:20:41Z",
        "kind": "event",
        "type": [
            "allowed",
            "connection"
        ]
    },
    "gcp": {
        "destination": {
            "instance": {
                "project_id": "test-beats",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "test-beats",
                "subnetwork_name": "windows-isolated",
                "vpc_name": "windows-isolated"
            }
        },
        "firewall": {
            "rule_details": {
                "action": "ALLOW",
                "direction": "INGRESS",
                "ip_port_info": [
                    {
                        "ip_protocol": "TCP",
                        "port_range": [
                            "3389"
                        ]
                    }
                ],
                "priority": 1000,
                "source_range": [
                    "0.0.0.0/0"
                ],
                "target_tag": [
                    "allow-rdp"
                ]
            }
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "logger": "projects/test-beats/logs/compute.googleapis.com%2Ffirewall"
    },
    "network": {
        "community_id": "1:OdLB9eXsBDLz8m97ao4LepX6q+4=",
        "direction": "inbound",
        "iana_number": "6",
        "name": "windows-isolated",
        "transport": "tcp",
        "type": "ipv4"
    },
    "related": {
        "ip": [
            "192.168.2.126",
            "10.42.0.2"
        ]
    },
    "rule": {
        "name": "network:windows-isolated/firewall:windows-isolated-allow-rdp"
    },
    "source": {
        "address": "192.168.2.126",
        "geo": {
            "continent_name": "Asia",
            "country_name": "omn"
        },
        "ip": "192.168.2.126",
        "port": 64853
    },
    "tags": [
        "forwarded",
        "gcp-firewall"
    ]
}
VPC Flow
edit

The vpcflow dataset collects logs sent from and received by VM instances, including instances used as GKE nodes.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.destination.instance.project_id

ID of the project containing the VM.

keyword

gcp.destination.instance.region

Region of the VM.

keyword

gcp.destination.instance.zone

Zone of the VM.

keyword

gcp.destination.vpc.project_id

ID of the project containing the VM.

keyword

gcp.destination.vpc.subnetwork_name

Subnetwork on which the VM is operating.

keyword

gcp.destination.vpc.vpc_name

VPC on which the VM is operating.

keyword

gcp.source.instance.project_id

ID of the project containing the VM.

keyword

gcp.source.instance.region

Region of the VM.

keyword

gcp.source.instance.zone

Zone of the VM.

keyword

gcp.source.vpc.project_id

ID of the project containing the VM.

keyword

gcp.source.vpc.subnetwork_name

Subnetwork on which the VM is operating.

keyword

gcp.source.vpc.vpc_name

VPC on which the VM is operating.

keyword

gcp.vpcflow.flattened

Contains the full vpcflow document as sent by GCP.

flattened

gcp.vpcflow.reporter

The side which reported the flow. Can be either SRC or DEST.

keyword

gcp.vpcflow.rtt.ms

Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.

long

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

Example

An example event for vpcflow looks as following:

{
    "@timestamp": "2019-06-14T03:50:10.845Z",
    "agent": {
        "ephemeral_id": "0b8165a2-0e25-4e9a-bb68-271697e0993f",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "availability_zone": "us-east1-b",
        "instance": {
            "name": "kibana"
        },
        "project": {
            "id": "my-sample-project"
        },
        "provider": "gcp",
        "region": "us-east1"
    },
    "data_stream": {
        "dataset": "gcp.vpcflow",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "10.139.99.242",
        "domain": "elasticsearch",
        "ip": "10.139.99.242",
        "port": 9200
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "created": "2023-10-25T04:21:42.006Z",
        "dataset": "gcp.vpcflow",
        "end": "2019-06-14T03:49:51.821056075Z",
        "id": "ut8lbrffooxz5",
        "ingested": "2023-10-25T04:21:43Z",
        "kind": "event",
        "start": "2019-06-14T03:40:20.510622432Z",
        "type": [
            "connection"
        ]
    },
    "gcp": {
        "destination": {
            "instance": {
                "project_id": "my-sample-project",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "my-sample-project",
                "subnetwork_name": "default",
                "vpc_name": "default"
            }
        },
        "source": {
            "instance": {
                "project_id": "my-sample-project",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "my-sample-project",
                "subnetwork_name": "default",
                "vpc_name": "default"
            }
        },
        "vpcflow": {
            "reporter": "DEST",
            "rtt": {
                "ms": 201
            }
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "logger": "projects/my-sample-project/logs/compute.googleapis.com%2Fvpc_flows"
    },
    "network": {
        "bytes": 11773,
        "community_id": "1:FYaJFSEAKLcBCMFoT6sR5TMHf/s=",
        "direction": "internal",
        "iana_number": "6",
        "name": "default",
        "packets": 94,
        "transport": "tcp",
        "type": "ipv4"
    },
    "related": {
        "ip": [
            "67.43.156.13",
            "10.139.99.242"
        ]
    },
    "source": {
        "address": "67.43.156.13",
        "as": {
            "number": 35908
        },
        "bytes": 11773,
        "domain": "kibana",
        "geo": {
            "continent_name": "Asia",
            "country_iso_code": "BT",
            "country_name": "Bhutan",
            "location": {
                "lat": 27.5,
                "lon": 90.5
            }
        },
        "ip": "67.43.156.13",
        "packets": 94,
        "port": 33576
    },
    "tags": [
        "forwarded",
        "gcp-vpcflow"
    ]
}
DNS
edit

The dns dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.dns.auth_answer

Authoritative answer.

boolean

gcp.dns.destination_ip

Destination IP address, only applicable for forwarding cases.

ip

gcp.dns.egress_error

Egress proxy error.

keyword

gcp.dns.flattened

Contains the full dns document as sent by GCP.

flattened

gcp.dns.protocol

Protocol TCP or UDP.

keyword

gcp.dns.query_name

DNS query name.

keyword

gcp.dns.query_type

DNS query type.

keyword

gcp.dns.rdata

DNS answer in presentation format, truncated to 260 bytes.

keyword

gcp.dns.response_code

Response code.

keyword

gcp.dns.server_latency

Server latency.

integer

gcp.dns.source_ip

Source IP address of the query.

ip

gcp.dns.source_network

Source network of the query.

keyword

gcp.dns.source_type

Type of source generating the DNS query: private-zone, public-zone, forwarding-zone, forwarding-policy, peering-zone, internal, external, internet

keyword

gcp.dns.target_type

Type of target resolving the DNS query: private-zone, public-zone, forwarding-zone, forwarding-policy, peering-zone, internal, external, internet

keyword

gcp.dns.vm_instance_id

Compute Engine VM instance ID, only applicable to queries initiated by Compute Engine VMs.

keyword

gcp.dns.vm_instance_name

Compute Engine VM instance name, only applicable to queries initiated by Compute Engine VMs.

keyword

gcp.dns.vm_project_id

Google Cloud project ID, only applicable to queries initiated by Compute Engine VMs.

keyword

gcp.dns.vm_zone_name

Google Cloud VM zone, only applicable to queries initiated by Compute Engine VMs.

keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

Example

An example event for dns looks as following:

{
    "@timestamp": "2021-12-12T15:59:40.446Z",
    "agent": {
        "ephemeral_id": "fd6c4189-cbc6-493a-acfb-c9e7b2b7588c",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "project": {
            "id": "key-reference-123456"
        },
        "provider": "gcp",
        "region": "global"
    },
    "data_stream": {
        "dataset": "gcp.dns",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "216.239.32.106",
        "ip": "216.239.32.106"
    },
    "dns": {
        "answers": [
            {
                "class": "IN",
                "data": "67.43.156.13",
                "name": "asdf.gcp.example.com.",
                "ttl": 300,
                "type": "A"
            }
        ],
        "question": {
            "name": "asdf.gcp.example.com",
            "registered_domain": "example.com",
            "subdomain": "asdf.gcp",
            "top_level_domain": "com",
            "type": "A"
        },
        "resolved_ip": [
            "67.43.156.13"
        ],
        "response_code": "NOERROR"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "dns-query",
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "created": "2023-10-25T04:19:40.300Z",
        "dataset": "gcp.dns",
        "id": "zir4wud11tm",
        "ingested": "2023-10-25T04:19:41Z",
        "kind": "event",
        "outcome": "success"
    },
    "gcp": {
        "dns": {
            "auth_answer": true,
            "destination_ip": "216.239.32.106",
            "protocol": "UDP",
            "query_name": "asdf.gcp.example.com.",
            "query_type": "A",
            "response_code": "NOERROR",
            "server_latency": 0,
            "source_type": "internet",
            "target_type": "public-zone"
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/key-reference-123456/logs/dns.googleapis.com%2Fdns_queries"
    },
    "network": {
        "iana_number": "17",
        "protocol": "dns",
        "transport": "udp"
    },
    "related": {
        "hosts": [
            "asdf.gcp.example.com"
        ],
        "ip": [
            "67.43.156.13",
            "216.239.32.106"
        ]
    },
    "tags": [
        "forwarded",
        "gcp-dns"
    ]
}
Loadbalancing Logs
edit

The loadbalancing_logs dataset collects logs of the requests sent to and handled by GCP Load Balancers.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.load_balancer.backend_service_name

The backend service to which the load balancer is sending traffic

keyword

gcp.load_balancer.cache_hit

Whether or not an entity was served from cache (with or without validation).

boolean

gcp.load_balancer.cache_id

Indicates the location and cache instance that the cache response was served from. For example, a cache response served from a cache in Amsterdam would have a cacheId value of AMS-85e2bd4b, where AMS is the IATA code, and 85e2bd4b is an opaque identifier of the cache instance (because some Cloud CDN locations have multiple discrete caches).

keyword

gcp.load_balancer.cache_lookup

Whether or not a cache lookup was attempted.

boolean

gcp.load_balancer.forwarding_rule_name

The name of the forwarding rule

keyword

gcp.load_balancer.status_details

Explains why the load balancer returned the HTTP status that it did. See https://cloud.google.com/cdn/docs/cdn-logging-monitoring#statusdetail_http_success_messages for specific messages.

keyword

gcp.load_balancer.target_proxy_name

The target proxy name

keyword

gcp.load_balancer.url_map_name

The URL map name

keyword

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

input.type

Input type

keyword

log.offset

Log offset

long

Example

An example event for loadbalancing looks as following:

{
    "@timestamp": "2020-06-08T23:41:30.078Z",
    "agent": {
        "ephemeral_id": "f4dde373-2ff7-464b-afdb-da94763f219b",
        "id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.6.0"
    },
    "cloud": {
        "project": {
            "id": "PROJECT_ID"
        },
        "region": "global"
    },
    "data_stream": {
        "dataset": "gcp.loadbalancing_logs",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "81.2.69.193",
        "ip": "81.2.69.193",
        "nat": {
            "ip": "10.5.3.1",
            "port": 9090
        },
        "port": 8080
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
        "snapshot": true,
        "version": "8.6.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "created": "2020-06-08T23:41:30.588Z",
        "dataset": "gcp.loadbalancing_logs",
        "id": "1oek5rg3l3fxj7",
        "ingested": "2023-01-13T15:02:22Z",
        "kind": "event",
        "type": [
            "info"
        ]
    },
    "gcp": {
        "load_balancer": {
            "backend_service_name": "",
            "cache_hit": true,
            "cache_id": "SFO-fbae48ad",
            "cache_lookup": true,
            "forwarding_rule_name": "FORWARDING_RULE_NAME",
            "status_details": "response_from_cache",
            "target_proxy_name": "TARGET_PROXY_NAME",
            "url_map_name": "URL_MAP_NAME"
        }
    },
    "http": {
        "request": {
            "bytes": 577,
            "method": "GET",
            "referrer": "https://developer.mozilla.org/en-US/docs/Web/JavaScript"
        },
        "response": {
            "bytes": 157,
            "status_code": 304
        },
        "version": "2.0"
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/PROJECT_ID/logs/requests"
    },
    "network": {
        "protocol": "http"
    },
    "related": {
        "ip": [
            "89.160.20.156",
            "81.2.69.193",
            "10.5.3.1"
        ]
    },
    "source": {
        "address": "89.160.20.156",
        "as": {
            "number": 29518,
            "organization": {
                "name": "Bredband2 AB"
            }
        },
        "geo": {
            "city_name": "Linköping",
            "continent_name": "Europe",
            "country_iso_code": "SE",
            "country_name": "Sweden",
            "location": {
                "lat": 58.4167,
                "lon": 15.6167
            },
            "region_iso_code": "SE-E",
            "region_name": "Östergötland County"
        },
        "ip": "89.160.20.156",
        "port": 9989
    },
    "tags": [
        "forwarded",
        "gcp-loadbalancing_logs"
    ],
    "url": {
        "domain": "81.2.69.193",
        "extension": "jpg",
        "original": "http://81.2.69.193:8080/static/us/three-cats.jpg",
        "path": "/static/us/three-cats.jpg",
        "port": 8080,
        "scheme": "http"
    },
    "user_agent": {
        "device": {
            "name": "Mac"
        },
        "name": "Chrome",
        "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36",
        "os": {
            "full": "Mac OS X 10.14.6",
            "name": "Mac OS X",
            "version": "10.14.6"
        },
        "version": "83.0.4103.61"
    }
}

Metrics

edit
Billing
edit

The billing dataset collects GCP Billing information from Google Cloud BigQuery daily cost detail table.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.billing.billing_account_id

Project Billing Account ID.

keyword

gcp.billing.cost_type

Cost types include regular, tax, adjustment, and rounding_error.

keyword

gcp.billing.effective_price

The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it’s the list price.

float

gcp.billing.invoice_month

Billing report month.

keyword

gcp.billing.project_id

Project ID of the billing report belongs to.

keyword

gcp.billing.project_name

Project Name of the billing report belongs to.

keyword

gcp.billing.service_description

The Google Cloud service that reported the Cloud Billing data.

keyword

gcp.billing.service_id

The ID of the service that the usage is associated with.

keyword

gcp.billing.sku_description

A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.

keyword

gcp.billing.sku_id

The ID of the resource used by the service.

keyword

gcp.billing.tags

A collection of key-value pairs that provide additional metadata.

nested

gcp.billing.tags.key

keyword

gcp.billing.tags.value

keyword

gcp.billing.total

Total billing amount.

float

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

Example

An example event for billing looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "01475F-5B1080-1137E7"
        },
        "project": {
            "id": "elastic-bi",
            "name": "elastic-containerlib-prod"
        },
        "provider": "gcp"
    },
    "event": {
        "dataset": "gcp.billing",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "billing": {
            "billing_account_id": "01475F-5B1080-1137E7",
            "cost_type": "regular",
            "invoice_month": "202106",
            "project_id": "containerlib-prod-12763",
            "project_name": "elastic-containerlib-prod",
            "total": 4717.170681,
            "sku_id": "0D56-2F80-52A5",
            "service_id": "6F81-5844-456A",
            "sku_description": "Network Inter Region Ingress from Jakarta to Americas",
            "service_description": "Compute Engine",
            "effective_price": 0.00292353,
            "tags": [
                {
                    "key": "stage",
                    "value": "prod"
                },
                {
                    "key": "size",
                    "value": "standard"
                }
            ]
        }
    },
    "metricset": {
        "name": "billing",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}
Compute
edit

The compute dataset is designed to fetch metrics for Compute Engine Virtual Machines in Google Cloud Platform.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Metric Type

@timestamp

Event timestamp.

date

agent.id

Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.

keyword

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.availability_zone

Availability zone in which this host, resource, or service is located.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

cloud.instance.id

Instance ID of the host machine.

keyword

cloud.machine.type

Machine type of the host machine.

keyword

cloud.region

Region in which this host, resource, or service is located.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.compute.firewall.dropped.bytes

Delta of incoming bytes dropped by the firewall

long

gauge

gcp.compute.firewall.dropped_packets_count.value

Delta of incoming packets dropped by the firewall

long

gauge

gcp.compute.instance.cpu.reserved_cores.value

Number of cores reserved on the host of the instance

double

gauge

gcp.compute.instance.cpu.usage.pct

The fraction of the allocated CPU that is currently in use on the instance

double

gauge

gcp.compute.instance.cpu.usage_time.sec

Delta of usage for all cores in seconds

double

gauge

gcp.compute.instance.disk.read.bytes

Delta of count of bytes read from disk

long

gauge

gcp.compute.instance.disk.read_ops_count.value

Delta of count of disk read IO operations

long

gauge

gcp.compute.instance.disk.write.bytes

Delta of count of bytes written to disk

long

gauge

gcp.compute.instance.disk.write_ops_count.value

Delta of count of disk write IO operations

long

gauge

gcp.compute.instance.memory.balloon.ram_size.value

The total amount of memory in the VM. This metric is only available for VMs that belong to the e2 family.

long

gauge

gcp.compute.instance.memory.balloon.ram_used.value

Memory currently used in the VM. This metric is only available for VMs that belong to the e2 family.

long

gauge

gcp.compute.instance.memory.balloon.swap_in.bytes

Delta of the amount of memory read into the guest from its own swap space. This metric is only available for VMs that belong to the e2 family.

long

gauge

gcp.compute.instance.memory.balloon.swap_out.bytes

Delta of the amount of memory written from the guest to its own swap space. This metric is only available for VMs that belong to the e2 family.

long

gauge

gcp.compute.instance.network.egress.bytes

Delta of count of bytes sent over the network

long

gauge

gcp.compute.instance.network.egress.packets.count

Delta of count of packets sent over the network

long

gauge

gcp.compute.instance.network.ingress.bytes

Delta of count of bytes received from the network

long

gauge

gcp.compute.instance.network.ingress.packets.count

Delta of count of packets received from the network

long

gauge

gcp.compute.instance.uptime.sec

Delta of number of seconds the VM has been running.

long

gauge

gcp.compute.instance.uptime_total.sec

Elapsed time since the VM was started, in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.

long

gauge

gcp.labels.metadata.*

object

gcp.labels.metrics.*

object

gcp.labels.resource.*

object

gcp.labels.system.*

object

gcp.labels.user.*

object

gcp.labels_fingerprint

Hashed value of the labels field.

keyword

gcp.metrics....

Metrics that returned from Google Cloud API query.

object

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

Example

An example event for compute looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.compute",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "compute": {
            "firewall": {
                "dropped": {
                    "bytes": 421
                },
                "dropped_packets_count": {
                    "value": 4
                }
            },
            "instance": {
                "cpu": {
                    "reserved_cores": {
                        "value": 1
                    },
                    "usage": {
                        "pct": 0.07259952346383708
                    },
                    "usage_time": {
                        "sec": 4.355971407830225
                    }
                },
                "memory": {
                    "balloon": {
                        "ram_size": {
                            "value": 4128378880
                        },
                        "ram_used": {
                            "value": 2190848000
                        },
                        "swap_in": {
                            "bytes": 0
                        },
                        "swap_out": {
                            "bytes": 0
                        }
                    }
                },
                "uptime": {
                    "sec": 60.00000000000091
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "compute",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}
Dataproc
edit

The dataproc dataset is designed to fetch metrics from Dataproc in Google Cloud Platform.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Metric Type

@timestamp

Event timestamp.

date

agent.id

Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.

keyword

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.dataproc.batch.spark.executors.count

Indicates the number of Batch Spark executors.

long

gauge

gcp.dataproc.cluster.hdfs.datanodes.count

Indicates the number of HDFS DataNodes that are running inside a cluster.

long

gauge

gcp.dataproc.cluster.hdfs.storage_capacity.value

Indicates capacity of HDFS system running on cluster in GB.

double

gauge

gcp.dataproc.cluster.hdfs.storage_utilization.value

The percentage of HDFS storage currently used.

double

gauge

gcp.dataproc.cluster.hdfs.unhealthy_blocks.count

Indicates the number of unhealthy blocks inside the cluster.

long

gauge

gcp.dataproc.cluster.job.completion_time.value

The time jobs took to complete from the time the user submits a job to the time Dataproc reports it is completed.

object

gcp.dataproc.cluster.job.duration.value

The time jobs have spent in a given state.

object

gcp.dataproc.cluster.job.failed.count

Indicates the delta of the number of jobs that have failed on a cluster.

long

gauge

gcp.dataproc.cluster.job.running.count

Indicates the number of jobs that are running on a cluster.

long

gauge

gcp.dataproc.cluster.job.submitted.count

Indicates the delta of the number of jobs that have been submitted to a cluster.

long

gauge

gcp.dataproc.cluster.operation.completion_time.value

The time operations took to complete from the time the user submits a operation to the time Dataproc reports it is completed.

object

gcp.dataproc.cluster.operation.duration.value

The time operations have spent in a given state.

object

gcp.dataproc.cluster.operation.failed.count

Indicates the delta of the number of operations that have failed on a cluster.

long

gauge

gcp.dataproc.cluster.operation.running.count

Indicates the number of operations that are running on a cluster.

long

gauge

gcp.dataproc.cluster.operation.submitted.count

Indicates the delta of the number of operations that have been submitted to a cluster.

long

gauge

gcp.dataproc.cluster.yarn.allocated_memory_percentage.value

The percentage of YARN memory is allocated.

double

gauge

gcp.dataproc.cluster.yarn.apps.count

Indicates the number of active YARN applications.

long

gauge

gcp.dataproc.cluster.yarn.containers.count

Indicates the number of YARN containers.

long

gauge

gcp.dataproc.cluster.yarn.memory_size.value

Indicates the YARN memory size in GB.

double

gauge

gcp.dataproc.cluster.yarn.nodemanagers.count

Indicates the number of YARN NodeManagers running inside cluster.

long

gauge

gcp.dataproc.cluster.yarn.pending_memory_size.value

The current memory request, in GB, that is pending to be fulfilled by the scheduler.

double

gauge

gcp.dataproc.cluster.yarn.virtual_cores.count

Indicates the number of virtual cores in YARN.

long

gauge

gcp.labels.metadata.*

object

gcp.labels.metrics.*

object

gcp.labels.resource.*

object

gcp.labels.system.*

object

gcp.labels.user.*

object

gcp.labels_fingerprint

Hashed value of the labels field.

keyword

gcp.metrics....

Metrics that returned from Google Cloud API query.

object

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

Example

An example event for dataproc looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.dataproc",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "dataproc": {
            "cluster": {
                "hdfs": {
                    "datanodes": {
                        "count": 15
                    }
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "dataproc",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}
Firestore
edit

The firestore dataset fetches metrics from Firestore in Google Cloud Platform.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Metric Type

@timestamp

Event timestamp.

date

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.firestore.document.delete.count

Delta of the number of successful document deletes.

long

gauge

gcp.firestore.document.read.count

Delta of the number of successful document reads from queries or lookups.

long

gauge

gcp.firestore.document.write.count

Delta of the number of successful document writes.

long

gauge

gcp.labels.metadata.*

object

gcp.labels.metrics.*

object

gcp.labels.resource.*

object

gcp.labels.system.*

object

gcp.labels.user.*

object

gcp.metrics....

Metrics that returned from Google Cloud API query.

object

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

Example

An example event for firestore looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.firestore",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "firestore": {
            "document": {
                "delete": {
                    "count": 3
                },
                "read": {
                    "count": 10
                },
                "write": {
                    "count": 1
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "firestore",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}
GKE
edit

The gke dataset is designed to fetch metrics from GKE in Google Cloud Platform.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Metric Type

@timestamp

Event timestamp.

date

agent.id

Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.

keyword

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.gke.container.cpu.core_usage_time.sec

Cumulative CPU usage on all cores used by the container in seconds. Sampled every 60 seconds.

double

counter

gcp.gke.container.cpu.limit_cores.value

CPU cores limit of the container. Sampled every 60 seconds.

double

gauge

gcp.gke.container.cpu.limit_utilization.pct

The fraction of the CPU limit that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed the limit. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.

double

gauge

gcp.gke.container.cpu.request_cores.value

Number of CPU cores requested by the container. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.

double

gauge

gcp.gke.container.cpu.request_utilization.pct

The fraction of the requested CPU that is currently in use on the instance. This value can be greater than 1 as usage can exceed the request. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.

double

gauge

gcp.gke.container.ephemeral_storage.limit.bytes

Local ephemeral storage limit in bytes. Sampled every 60 seconds.

long

gauge

gcp.gke.container.ephemeral_storage.request.bytes

Local ephemeral storage request in bytes. Sampled every 60 seconds.

long

gauge

gcp.gke.container.ephemeral_storage.used.bytes

Local ephemeral storage usage in bytes. Sampled every 60 seconds.

long

gauge

gcp.gke.container.memory.limit.bytes

Memory limit of the container in bytes. Sampled every 60 seconds.

long

gauge

gcp.gke.container.memory.limit_utilization.pct

The fraction of the memory limit that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed the limit. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.

double

gauge

gcp.gke.container.memory.page_fault.count

Number of page faults, broken down by type, major and minor.

long

counter

gcp.gke.container.memory.request.bytes

Memory request of the container in bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.

long

gauge

gcp.gke.container.memory.request_utilization.pct

The fraction of the requested memory that is currently in use on the instance. This value can be greater than 1 as usage can exceed the request. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.

double

gauge

gcp.gke.container.memory.used.bytes

Memory usage in bytes. Sampled every 60 seconds.

long

gauge

gcp.gke.container.restart.count

Number of times the container has restarted. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.

long

counter

gcp.gke.container.uptime.sec

Time in seconds that the container has been running. Sampled every 60 seconds.

double

gauge

gcp.gke.node.cpu.allocatable_cores.value

Number of allocatable CPU cores on the node. Sampled every 60 seconds.

double

gauge

gcp.gke.node.cpu.allocatable_utilization.pct

The fraction of the allocatable CPU that is currently in use on the instance. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.

double

gauge

gcp.gke.node.cpu.core_usage_time.sec

Cumulative CPU usage on all cores used on the node in seconds. Sampled every 60 seconds.

double

counter

gcp.gke.node.cpu.total_cores.value

Total number of CPU cores on the node. Sampled every 60 seconds.

double

gauge

gcp.gke.node.ephemeral_storage.allocatable.bytes

Local ephemeral storage bytes allocatable on the node. Sampled every 60 seconds.

long

gauge

gcp.gke.node.ephemeral_storage.inodes_free.value

Free number of inodes on local ephemeral storage. Sampled every 60 seconds.

long

gauge

gcp.gke.node.ephemeral_storage.inodes_total.value

Total number of inodes on local ephemeral storage. Sampled every 60 seconds.

long

gauge

gcp.gke.node.ephemeral_storage.total.bytes

Total ephemeral storage bytes on the node. Sampled every 60 seconds.

long

gauge

gcp.gke.node.ephemeral_storage.used.bytes

Local ephemeral storage bytes used by the node. Sampled every 60 seconds.

long

gauge

gcp.gke.node.memory.allocatable.bytes

Cumulative memory bytes used by the node. Sampled every 60 seconds.

long

gauge

gcp.gke.node.memory.allocatable_utilization.pct

The fraction of the allocatable memory that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed allocatable memory bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.

double

gauge

gcp.gke.node.memory.total.bytes

Number of bytes of memory allocatable on the node. Sampled every 60 seconds.

long

gauge

gcp.gke.node.memory.used.bytes

Cumulative memory bytes used by the node. Sampled every 60 seconds.

long

gauge

gcp.gke.node.network.received_bytes.count

Cumulative number of bytes received by the node over the network. Sampled every 60 seconds.

long

counter

gcp.gke.node.network.sent_bytes.count

Cumulative number of bytes transmitted by the node over the network. Sampled every 60 seconds.

long

counter

gcp.gke.node.pid_limit.value

The max PID of OS on the node. Sampled every 60 seconds.

long

gauge

gcp.gke.node.pid_used.value

The number of running process in the OS on the node. Sampled every 60 seconds.

long

gauge

gcp.gke.node_daemon.cpu.core_usage_time.sec

Cumulative CPU usage on all cores used by the node level system daemon in seconds. Sampled every 60 seconds.

double

counter

gcp.gke.node_daemon.memory.used.bytes

Memory usage by the system daemon in bytes. Sampled every 60 seconds.

long

gauge

gcp.gke.pod.network.received.bytes

Cumulative number of bytes received by the pod over the network. Sampled every 60 seconds.

long

counter

gcp.gke.pod.network.sent.bytes

Cumulative number of bytes transmitted by the pod over the network. Sampled every 60 seconds.

long

counter

gcp.gke.pod.volume.total.bytes

Total number of disk bytes available to the pod. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.

long

gauge

gcp.gke.pod.volume.used.bytes

Number of disk bytes used by the pod. Sampled every 60 seconds.

long

gauge

gcp.gke.pod.volume.utilization.pct

The fraction of the volume that is currently being used by the instance. This value cannot be greater than 1 as usage cannot exceed the total available volume space. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.

double

gauge

gcp.labels.metadata.*

object

gcp.labels.metrics.*

object

gcp.labels.resource.*

object

gcp.labels.system.*

object

gcp.labels.user.*

object

gcp.labels_fingerprint

Hashed value of the labels field.

keyword

gcp.metrics....

Metrics that returned from Google Cloud API query.

object

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

Example

An example event for gke looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.gke",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "gke": {
            "container": {
                "cpu": {
                    "core_usage_time": {
                        "sec": 15
                    }
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "gke",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}
Loadbalancing Metrics
edit

The loadbalancing_metrics dataset is designed to fetch HTTPS, HTTP, and Layer 3 metrics from Load Balancing in Google Cloud Platform.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Metric Type

@timestamp

Event timestamp.

date

agent.id

Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.

keyword

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.labels.metadata.*

object

gcp.labels.metrics.*

object

gcp.labels.resource.*

object

gcp.labels.system.*

object

gcp.labels.user.*

object

gcp.labels_fingerprint

Hashed value of the labels field.

keyword

gcp.loadbalancing_metrics.https.backend_latencies.value

A distribution of the latency calculated from when the request was sent by the proxy to the backend until the proxy received from the backend the last byte of response.

object

gcp.loadbalancing_metrics.https.backend_request.bytes

Delta of the number of bytes sent as requests from HTTP/S load balancer to backends.

long

gauge

gcp.loadbalancing_metrics.https.backend_request.count

Delta of the number of requests served by backends of HTTP/S load balancer.

long

gauge

gcp.loadbalancing_metrics.https.backend_response.bytes

Delta of the number of bytes sent as responses from backends (or cache) to external HTTP(S) load balancer.

long

gauge

gcp.loadbalancing_metrics.https.external.regional.backend_latencies.value

A distribution of the latency calculated from when the request was sent by the proxy to the backend until the proxy received from the backend the last byte of response.

object

gcp.loadbalancing_metrics.https.external.regional.total_latencies.value

A distribution of the latency calculated from when the request was received by the proxy until the proxy got ACK from client on last response byte.

object

gcp.loadbalancing_metrics.https.frontend_tcp_rtt.value

A distribution of the RTT measured for each connection between client and proxy.

object

gcp.loadbalancing_metrics.https.internal.backend_latencies.value

A distribution of the latency calculated from when the request was sent by the internal HTTP/S load balancer proxy to the backend until the proxy received from the backend the last byte of response.

object

gcp.loadbalancing_metrics.https.internal.total_latencies.value

A distribution of the latency calculated from when the request was received by the internal HTTP/S load balancer proxy until the proxy got ACK from client on last response byte.

object

gcp.loadbalancing_metrics.https.request.bytes

Delta of the number of bytes sent as requests from clients to HTTP/S load balancer.

long

gauge

gcp.loadbalancing_metrics.https.request.count

Delta of the number of requests served by HTTP/S load balancer.

long

gauge

gcp.loadbalancing_metrics.https.response.bytes

Delta of the number of bytes sent as responses from HTTP/S load balancer to clients.

long

gauge

gcp.loadbalancing_metrics.https.total_latencies.value

A distribution of the latency calculated from when the request was received by the external HTTP/S load balancer proxy until the proxy got ACK from client on last response byte.

object

gcp.loadbalancing_metrics.l3.external.egress.bytes

Delta of the number of bytes sent from external TCP/UDP network load balancer backend to client of the flow. For TCP flows it’s counting bytes on application stream only.

long

gauge

gcp.loadbalancing_metrics.l3.external.egress_packets.count

Delta of the number of packets sent from external TCP/UDP network load balancer backend to client of the flow.

long

gauge

gcp.loadbalancing_metrics.l3.external.ingress.bytes

Delta of the number of bytes sent from client to external TCP/UDP network load balancer backend. For TCP flows it’s counting bytes on application stream only.

long

gauge

gcp.loadbalancing_metrics.l3.external.ingress_packets.count

Delta of the number of packets sent from client to external TCP/UDP network load balancer backend.

long

gauge

gcp.loadbalancing_metrics.l3.external.rtt_latencies.value

A distribution of the round trip time latency, measured over TCP connections for the external network load balancer.

object

gcp.loadbalancing_metrics.l3.internal.egress.bytes

Delta of the number of bytes sent from ILB backend to client (for TCP flows it’s counting bytes on application stream only).

long

gauge

gcp.loadbalancing_metrics.l3.internal.egress_packets.count

Delta of the number of packets sent from ILB backend to client of the flow.

long

gauge

gcp.loadbalancing_metrics.l3.internal.ingress.bytes

Delta of the number of bytes sent from client to ILB backend (for TCP flows it’s counting bytes on application stream only).

long

gauge

gcp.loadbalancing_metrics.l3.internal.ingress_packets.count

Delta of the number of packets sent from client to ILB backend.

long

gauge

gcp.loadbalancing_metrics.l3.internal.rtt_latencies.value

A distribution of RTT measured over TCP connections for internal TCP/UDP load balancer flows.

object

gcp.loadbalancing_metrics.tcp_ssl_proxy.closed_connections.value

Delta of the number of connections that were terminated over TCP/SSL proxy.

long

gauge

gcp.loadbalancing_metrics.tcp_ssl_proxy.egress.bytes

Delta of the number of bytes sent from VM to client using proxy.

long

gauge

gcp.loadbalancing_metrics.tcp_ssl_proxy.frontend_tcp_rtt.value

A distribution of the smoothed RTT (in ms) measured by the proxy’s TCP stack, each minute application layer bytes pass from proxy to client.

object

gcp.loadbalancing_metrics.tcp_ssl_proxy.ingress.bytes

Delta of the number of bytes sent from client to VM using proxy.

long

gauge

gcp.loadbalancing_metrics.tcp_ssl_proxy.new_connections.value

Delta of the number of connections that were created over TCP/SSL proxy.

long

gauge

gcp.loadbalancing_metrics.tcp_ssl_proxy.open_connections.value

Current number of outstanding connections through the TCP/SSL proxy.

long

gauge

gcp.metrics....

Metrics that returned from Google Cloud API query.

object

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

Example

An example event for loadbalancing looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-observability"
        },
        "provider": "gcp",
        "region": "us-central1",
        "availability_zone": "us-central1-a"
    },
    "event": {
        "dataset": "gcp.loadbalancing_metrics",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "labels": {
            "metrics": {
                "client_network": "ocp-be-c5kjr-network",
                "client_subnetwork": "ocp-be-c5kjr-worker-subnet",
                "client_zone": "us-central1-a"
            },
            "resource": {
                "backend_name": "ocp-be-c5kjr-master-us-central1-a",
                "backend_scope": "us-central1-a",
                "backend_scope_type": "ZONE",
                "backend_subnetwork_name": "ocp-be-c5kjr-master-subnet",
                "backend_target_name": "ocp-be-c5kjr-api-internal",
                "backend_target_type": "BACKEND_SERVICE",
                "backend_type": "INSTANCE_GROUP",
                "forwarding_rule_name": "ocp-be-c5kjr-api-internal",
                "load_balancer_name": "ocp-be-c5kjr-api-internal",
                "network_name": "ocp-be-c5kjr-network",
                "region": "us-central1"
            }
        },
        "loadbalancing_metrics": {
            "l3": {
                "internal": {
                    "egress_packets": {
                        "count": 100
                    },
                    "egress": {
                        "bytes": 1247589
                    }
                }
            }
        }
    },
    "metricset": {
        "name": "loadbalancing",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}
Redis
edit

The redis dataset is designed to fetch metrics from GCP Memorystore for Redis in Google Cloud Platform.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Unit Metric Type

@timestamp

Event timestamp.

date

agent.id

Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.

keyword

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

cloud.instance.id

Instance ID of the host machine.

keyword

cloud.instance.name

Instance name of the host machine.

keyword

cloud.machine.type

Machine type of the host machine.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.labels.metadata.*

object

gcp.labels.metrics.*

object

gcp.labels.resource.*

object

gcp.labels.system.*

object

gcp.labels.user.*

object

gcp.labels_fingerprint

Hashed value of the labels field.

keyword

gcp.metrics....

Metrics that returned from Google Cloud API query.

object

gcp.redis.clients.blocked.count

Number of blocked clients.

long

gauge

gcp.redis.clients.connected.count

Number of client connections.

long

gauge

gcp.redis.commands.calls.count

Delta of the number of calls for this command in one minute.

long

gauge

gcp.redis.commands.total_time.us

Delta of the amount of time in microseconds that this command took in the last second.

long

micros

gauge

gcp.redis.commands.usec_per_call.sec

Average time per call over 1 minute by command.

double

s

gauge

gcp.redis.keyspace.avg_ttl.sec

Average TTL for keys in this database.

double

s

gauge

gcp.redis.keyspace.keys.count

Number of keys stored in this database.

long

gauge

gcp.redis.keyspace.keys_with_expiration.count

Number of keys with an expiration in this database.

long

gauge

gcp.redis.persistence.rdb.bgsave_in_progress

Flag indicating a RDB save is on-going.

boolean

gcp.redis.replication.master.slaves.lag.sec

The number of seconds that replica is lagging behind primary.

long

s

gauge

gcp.redis.replication.master.slaves.offset.bytes

The number of bytes that have been acknowledged by replicas.

long

byte

gauge

gcp.redis.replication.master_repl_offset.bytes

The number of bytes that master has produced and sent to replicas.

long

byte

gauge

gcp.redis.replication.offset_diff.bytes

The largest number of bytes that have not been replicated across all replicas. This is the biggest difference between replication byte offset (master) and replication byte offset (replica) of all replicas.

long

byte

gauge

gcp.redis.replication.role

Returns a value indicating the node role. 1 indicates primary and 0 indicates replica.

long

gauge

gcp.redis.server.uptime.sec

Uptime in seconds.

long

s

gauge

gcp.redis.stats.cache_hit_ratio

Cache Hit ratio as a fraction.

double

gauge

gcp.redis.stats.connections.total.count

Delta of the total number of connections accepted by the server.

long

gauge

gcp.redis.stats.cpu_utilization.sec

CPU-seconds consumed by the Redis server, broken down by system/user space and parent/child relationship.

double

s

gauge

gcp.redis.stats.evicted_keys.count

Delta of the number of evicted keys due to maxmemory limit.

long

gauge

gcp.redis.stats.expired_keys.count

Delta of the total number of key expiration events.

long

gauge

gcp.redis.stats.keyspace_hits.count

Delta of the number of successful lookup of keys in the main dictionary.

long

gauge

gcp.redis.stats.keyspace_misses.count

Delta of the number of failed lookup of keys in the main dictionary.

long

gauge

gcp.redis.stats.memory.maxmemory.mb

Maximum amount of memory Redis can consume.

long

m

gauge

gcp.redis.stats.memory.system_memory_overload_duration.us

The amount of time in microseconds the instance is in system memory overload mode.

long

micros

gauge

gcp.redis.stats.memory.system_memory_usage_ratio

Memory usage as a ratio of maximum system memory.

double

gauge

gcp.redis.stats.memory.usage.bytes

Total number of bytes allocated by Redis.

long

byte

gauge

gcp.redis.stats.memory.usage_ratio

Memory usage as a ratio of maximum memory.

double

gauge

gcp.redis.stats.network_traffic.bytes

Delta of the total number of bytes sent to/from redis (includes bytes from commands themselves, payload data, and delimiters).

long

byte

gauge

gcp.redis.stats.pubsub.channels.count

Global number of pub/sub channels with client subscriptions.

long

gauge

gcp.redis.stats.pubsub.patterns.count

Global number of pub/sub pattern with client subscriptions.

long

gauge

gcp.redis.stats.reject_connections.count

Number of connections rejected because of maxclients limit.

long

gauge

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

Example

An example event for redis looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.redis",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "redis": {
            "clients": {
                "blocked": {
                    "count": 4
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "metrics",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}
Storage
edit

The storage dataset fetches metrics from Storage in Google Cloud Platform.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Metric Type

@timestamp

Event timestamp.

date

agent.id

Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.

keyword

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.image.id

Image ID for the cloud instance.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

event.dataset

Event dataset

constant_keyword

event.module

Event module

constant_keyword

gcp.labels.metadata.*

object

gcp.labels.metrics.*

object

gcp.labels.resource.*

object

gcp.labels.system.*

object

gcp.labels.user.*

object

gcp.labels_fingerprint

Hashed value of the labels field.

keyword

gcp.metrics....

Metrics that returned from Google Cloud API query.

object

gcp.storage.api.request.count

Delta count of API calls, grouped by the API method name and response code.

long

gauge

gcp.storage.authz.acl_based_object_access.count

Delta count of requests that result in an object being granted access solely due to object ACLs.

long

gauge

gcp.storage.authz.acl_operations.count

Usage of ACL operations broken down by type.

long

gauge

gcp.storage.authz.object_specific_acl_mutation.count

Delta count of changes made to object specific ACLs.

long

gauge

gcp.storage.network.received.bytes

Delta count of bytes received over the network, grouped by the API method name and response code.

long

gauge

gcp.storage.network.sent.bytes

Delta count of bytes sent over the network, grouped by the API method name and response code.

long

gauge

gcp.storage.storage.object.count

Total number of objects per bucket, grouped by storage class. This value is measured once per day, and the value is repeated at each sampling interval throughout the day.

long

gauge

gcp.storage.storage.total.bytes

Total size of all objects in the bucket, grouped by storage class. This value is measured once per day, and the value is repeated at each sampling interval throughout the day.

long

gauge

gcp.storage.storage.total_byte_seconds.bytes

Delta count of bytes received over the network, grouped by the API method name and response code.

long

gauge

host.containerized

If the host is a container.

boolean

host.os.build

OS build information.

keyword

host.os.codename

OS codename, if any.

keyword

Example

An example event for storage looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.storage",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "storage": {
            "storage": {
                "total": {
                    "bytes": 4472520191
                }
            },
            "network": {
                "received": {
                    "bytes": 4472520191
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "storage",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Changelog

edit
Changelog
Version Details Kibana version(s)

2.38.0

Enhancement (View pull request)
Add policy_violation_info, metadata and related fields to audit logs.

Bug fix (View pull request)
Update GCP audit log dashboard to use correct email field.

8.13.0 or higher

2.37.2

Bug fix (View pull request)
Fix definition of subfields of nested objects

8.13.0 or higher

2.37.1

Enhancement (View pull request)
Improve GCP Billing documentation.

8.13.0 or higher

2.37.0

Enhancement (View pull request)
Retain authenticationInfo.serviceAccountKeyName data.

8.13.0 or higher

2.36.0

Enhancement (View pull request)
Add global dataset filter for dashboards to improve performance.

8.13.0 or higher

2.35.0

Enhancement (View pull request)
ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

2.34.1

Bug fix (View pull request)
Fix Redis metric type for persistence.rdb.bgsave_in_progress. Metric type should be boolean instead of long.

8.12.0 or higher

2.34.0

Enhancement (View pull request)
Add tags and processors to GCP Compute, Firestore, PostgreSQL.

8.12.0 or higher

2.33.2

Enhancement (View pull request)
Add tags and processors to GCP Storage

8.12.0 or higher

2.33.1

Enhancement (View pull request)
Update Legacy metric visualization to new metric in GCP Billing overview dashboard.

8.12.0 or higher

2.33.0

Enhancement (View pull request)
Enable time series data for metrics data streams. This dramatically reduces storage for metrics and is expected to progressively improve query [performance](https://www.elastic.co/blog/70-percent-storage-savings-for-metrics-with-elastic-observability). For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html.

8.12.0 or higher

2.32.1

Enhancement (View pull request)
Add dimensions mappings and the metrics_fingerprint field across all metrics data streams.

8.12.0 or higher

2.32.0

Enhancement (View pull request)
Add new billing data stream fields.

8.12.0 or higher

2.31.2

Bug fix (View pull request)
Fix pipeline error parsing DNS logs with empty rdata field.

8.7.1 or higher

2.31.1

Enhancement (View pull request)
Add Cloud Run docs and fix policy template name to allow adding Cloud Run logs to the policy.

8.7.1 or higher

2.31.0

Enhancement (View pull request)
Allow users to retain otherwised discarded fields.

8.7.1 or higher

2.30.1

Bug fix (View pull request)
Fix mappings of group fields

8.7.1 or higher

2.30.0

Enhancement (View pull request)
Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

Enhancement (View pull request)
Upgrade package spec to 3.0.0.

Bug fix (View pull request)
Fix orphan dashboard references.

Bug fix (View pull request)
Add missing dashboard filters.

8.7.1 or higher

2.29.1

Bug fix (View pull request)
Add null checks and ignore_missing checks to the rename processor

8.7.1 or higher

2.29.0

Bug fix (View pull request)
Remove GCP CloudSQL deprecated, alpha or beta metrics and fix field types.

8.7.1 or higher

2.28.5

Enhancement (View pull request)
Set metric type for GKE, Load Balancing, PubSub, Redis and Storage data streams.

8.7.1 or higher

2.28.4

Enhancement (View pull request)
Migrate GCP Load Balancing HTTPS Overview dashboard to lens.

8.7.1 or higher

2.28.3

Enhancement (View pull request)
Set metric type for Cloud Run, Compute, Dataproc and Firestore data streams.

8.7.1 or higher

2.28.2

Enhancement (View pull request)
Migrate GCP Load Balancing TCP SSL Proxy Overview dashboard to lens.

8.7.1 or higher

2.28.1

Enhancement (View pull request)
Set metric type for CloudSQL data streams.

8.7.1 or higher

2.28.0

Enhancement (View pull request)
Migrate GCP Load Balancing L3 Overview dashboard to lens.

8.7.1 or higher

2.27.0

Enhancement (View pull request)
Add GCP CloudSQL MySQL, SQL Server and PostgreSQL dashboards.

8.7.1 or higher

2.26.0

Bug fix (View pull request)
Fix GCP loadbalancing_metrics fields prefix.

8.7.1 or higher

2.25.1

Bug fix (View pull request)
Fix check on gcp.audit.authorization_info[].granted.

8.7.1 or higher

2.25.0

Enhancement (View pull request)
Migrate GCP Billing input control to new control panel.

8.7.1 or higher

2.24.0

Enhancement (View pull request)
Add GCP CloudSQL MySQL, Postgres, SQLServer data streams

8.7.1 or higher

2.23.0

Enhancement (View pull request)
Convert security dashboards to lens.

8.7.1 or higher

2.22.1

Enhancement (View pull request)
Change ownership in manifest.

8.6.0 or higher

2.22.0

Enhancement (View pull request)
Ensure event.kind is correctly set for pipeline errors.

8.6.0 or higher

2.21.0

Enhancement (View pull request)
Update package to ECS 8.8.0.

8.6.0 or higher

2.20.1

Bug fix (View pull request)
Fix invalid TSDS metric type for persistence.rdb.bgsave_in_progress field

8.6.0 or higher

2.20.0

Enhancement (View pull request)
Update package to ECS 8.7.0.

8.6.0 or higher

2.19.1

Enhancement (View pull request)
Migrate compute dashboard to lens and add datastream filter.

8.6.0 or higher

2.19.0

Enhancement (View pull request)
Add Cloud Run metrics datastream.

8.6.0 or higher

2.18.0

Enhancement (View pull request)
Support subscription_num_goroutines and subscription_max_outstanding_messages for GCP PubSub input

8.6.0 or higher

2.17.2

Bug fix (View pull request)
Fix IP Convert processor in Audit ingest pipeline.

8.6.0 or higher

2.17.1

Enhancement (View pull request)
Added categories and/or subcategories.

8.6.0 or higher

2.17.0

Enhancement (View pull request)
Add Audit Log Overview dashboard

Enhancement (View pull request)
Add GKE Overview dashboard

Enhancement (View pull request)
Add PubSub Overview dashboard

Enhancement (View pull request)
Add Storage Overview dashboard

8.6.0 or higher

2.16.2

Bug fix (View pull request)
Add logic to handle scalar request.policy values on audit

8.5.0 or higher

2.16.1

Bug fix (View pull request)
Replace missing input control panel with new-style control.

8.5.0 or higher

2.16.0

Enhancement (View pull request)
Update package to ECS 8.6.0.

8.5.0 or higher

2.15.2

Enhancement (View pull request)
Update documentation.

8.5.0 or higher

2.15.1

Enhancement (View pull request)
Add GCP Compute pipeline test.

8.5.0 or higher

2.15.0

Enhancement (View pull request)
Remove support for Kibana 7.17.x

Enhancement (View pull request)
Support multiple regions for metrics data streams

8.5.0 or higher

2.14.0

Enhancement (View pull request)
Update package to ECS 8.5.0.

8.3.0 or higher

2.13.0

Enhancement (View pull request)
Migrate dashboard by values

8.3.0 or higher

2.12.1

Bug fix (View pull request)
Remove duplicate fields.

7.17.6 or higher
8.3.0 or higher

2.12.0

Enhancement (View pull request)
Add GCP Redis

7.17.6 or higher
8.3.0 or higher

2.11.12

Bug fix (View pull request)
Add GKE ingest pipeline.

7.17.6 or higher
8.3.0 or higher

2.11.11

Bug fix (View pull request)
Fix type of dns.answers.ttl.

7.17.6 or higher
8.3.0 or higher

2.11.10

Enhancement (View pull request)
Add ingest pipeline for dataproc.

Enhancement (View pull request)
Add GCP loadbalancing ingest pipeline

Enhancement (View pull request)
Add GCP PubSub ingest pipeline

Enhancement (View pull request)
Add GCP Storage ingest pipeline

Enhancement (View pull request)
Add GCP Firestore ingest pipeline

Enhancement (View pull request)
Add GCP Compute ingest pipeline

7.17.6 or higher
8.3.0 or higher

2.11.10-beta.6

Enhancement (View pull request)
Add ingest pipeline for dataproc.

2.11.10-beta.5

Enhancement (View pull request)
Add GCP loadbalancing ingest pipeline

2.11.10-beta.4

Enhancement (View pull request)
Add GCP PubSub ingest pipeline

2.11.10-beta.3

Enhancement (View pull request)
Add GCP Storage ingest pipeline

2.11.10-beta.2

Enhancement (View pull request)
Add GCP Firestore ingest pipeline

2.11.10-beta.1

Enhancement (View pull request)
Add GCP Compute ingest pipeline

2.11.9

Bug fix (View pull request)
Fix GKE kubernetes.io indentation.

7.17.6 or higher
8.3.0 or higher

2.11.8

Enhancement (View pull request)
Remove duplicate fields.

7.17.6 or higher
8.3.0 or higher

2.11.7

Enhancement (View pull request)
Move Dataproc lightweight module config into integration

7.17.6 or higher
8.3.0 or higher

2.11.6

Enhancement (View pull request)
Move LoadBalancing lightweight module config into integration

7.17.6 or higher
8.3.0 or higher

2.11.5

Enhancement (View pull request)
Move Storage lightweight module config into integration

7.17.6 or higher
8.3.0 or higher

2.11.4

Enhancement (View pull request)
Move PubSub lightweight module config into integration

7.17.6 or higher
8.3.0 or higher

2.11.3

Enhancement (View pull request)
Move GKE lightweight module config into integration

7.17.6 or higher
8.3.0 or higher

2.11.2

Enhancement (View pull request)
Move Firestore lightweight module config into integration

7.17.6 or higher
8.3.0 or higher

2.11.1

Enhancement (View pull request)
Use ECS geo.location definition.

7.17.6 or higher
8.3.0 or higher

2.11.0

Enhancement (View pull request)
Move Compute lightweight module config into integration

7.17.6 or higher
8.3.0 or higher

2.10.0

Enhancement (View pull request)
Add GCP PubSub Data stream

7.17.6 or higher
8.3.0 or higher

2.9.0

Enhancement (View pull request)
Add GCP Dataproc Data stream

7.17.6 or higher
8.3.0 or higher

2.8.0

Enhancement (View pull request)
Add GCP GKE Data Stream

7.17.6 or higher
8.3.0 or higher

2.7.0

Enhancement (View pull request)
Add GCP Storage Data Stream

7.17.6 or higher
8.3.0 or higher

2.6.0

Enhancement (View pull request)
Add Load Balancing logs datastream

7.17.6 or higher
8.3.0 or higher

2.5.0

Enhancement (View pull request)
Add GCP Load Balancing Metricset

Bug fix (View pull request)
Fix credentials_json escaping in loadbalancing_metrics

Bug fix (View pull request)
Update loadbalancing_metrics default period to 60s

Bug fix (View pull request)
Fix event.dataset for loadbalancing_metrics

Enhancement (View pull request)
Add loadbalancing_metrics distribution fields

7.17.6 or higher
8.3.0 or higher

2.4.0

Enhancement (View pull request)
Update package to ECS 8.4.0

7.17.6 or higher
8.3.0 or higher

2.3.0

Enhancement (View pull request)
Add additional parsing for DNS Public Zone Query Logs

7.17.6 or higher
8.3.0 or higher

2.2.1

Enhancement (View pull request)
Fix Billing policy template title and default period for gcp.compute

7.17.6 or higher
8.3.0 or higher

2.2.0

Enhancement (View pull request)
Remove fields duplicated in ECS fields

7.17.6 or higher
8.3.0 or higher

2.1.0

Enhancement (foobar[View pull request])
restore compatibility with 7.17 release track

7.17.6 or higher
8.3.0 or higher

2.0.0

Breaking change (View pull request)
Move configurations to support metrics. This change is breaking, as it moves some configuration from the top level variables to data stream variables.

This change involves project_id, credentials_file and credentials_json variables that are moved from input level configuration to package level configuration (as those variables are reused across all inputs/data streams).

Users with GCP integration enabled will need to input values for these variables again when upgrading the policies to this version.

Enhancement (View pull request)
Add GCP Billing Data Stream

Enhancement (View pull request)
Add GCP Compute Data Stream

Enhancement (View pull request)
Add GCP Firestore Data stream

8.3.0 or higher

1.10.0

Enhancement (View pull request)
Update package to ECS 8.3.0.

7.17.0 or higher
8.0.0 or higher

1.9.2

Bug fix (View pull request)
Fix GCP auditlog parsing issue on response status

7.17.0 or higher
8.0.0 or higher

1.9.1

Enhancement (View pull request)
Update readme

7.17.0 or higher
8.0.0 or higher

1.9.0

Enhancement (View pull request)
Preserve request and response in flattened fields.

7.17.0 or higher
8.0.0 or higher

1.8.0

Enhancement (View pull request)
Add missing cloud.provider field.

7.17.0 or higher
8.0.0 or higher

1.7.0

Enhancement (View pull request)
Add dashboards for firewall and vpc flow logs.

Bug fix (View pull request)
Add missing mappings for several event.* fields.

1.6.1

Enhancement (View pull request)
Clarify the GCP privileges required by the Pub/Sub input.

7.16.3 or higher
8.0.0 or higher

1.6.0

Enhancement (View pull request)
Update to ECS 8.2

1.5.1

Enhancement (View pull request)
Add documentation for multi-fields

7.16.3 or higher
8.0.0 or higher

1.5.0

Enhancement (View pull request)
Improve Google Cloud Platform docs.

7.16.3 or higher
8.0.0 or higher

1.4.2

Bug fix (View pull request)
Remove emtpy values, names with only dots, and invalid client IPs.

7.16.3 or higher
8.0.0 or higher

1.4.1

Bug fix (View pull request)
Fix quoting of the credentials_json value in policy templates.

7.16.3 or higher
8.0.0 or higher

1.4.0

Enhancement (View pull request)
Add gcp.dns integration

1.3.1

Bug fix (View pull request)
Add Ingest Pipeline script to map IANA Protocol Numbers

7.15.0 or higher
8.0.0 or higher

1.3.0

Enhancement (View pull request)
Update to ECS 8.0

7.15.0 or higher
8.0.0 or higher

1.2.2

Bug fix (View pull request)
Regenerate test files using the new GeoIP database

7.15.0 or higher
8.0.0 or higher

1.2.1

Bug fix (View pull request)
Change test public IPs to the supported subset

1.2.0

Enhancement (View pull request)
Add 8.0.0 version constraint

7.15.0 or higher
8.0.0 or higher

1.1.2

Enhancement (View pull request)
Update Title and Description.

7.15.0 or higher

1.1.1

Bug fix (View pull request)
Fix logic that checks for the forwarded tag

1.1.0

Enhancement (View pull request)
Update to ECS 1.12.0

7.15.0 or higher

1.0.0

Enhancement (View pull request)
Move from experimental to GA

Enhancement (View pull request)
remove experimental from data_sets

0.3.3

Enhancement (View pull request)
Convert to generated ECS fields

0.3.2

Enhancement (View pull request)
update to ECS 1.11.0

0.3.1

Enhancement (View pull request)
Escape special characters in docs

0.3.0

Enhancement (View pull request)
Update integration description

0.2.0

Enhancement (View pull request)
Set "event.module" and "event.dataset"

0.1.0

Enhancement (View pull request)
update to ECS 1.10.0 and adding event.original options

0.0.2

Enhancement (View pull request)
update to ECS 1.9.0

0.0.1

Enhancement (View pull request)
initial release