Billing
editBilling
editVersion |
2.38.0 (View all) |
Compatible Kibana version(s) |
8.13.0 or higher |
Supported Serverless project types |
Security |
Subscription level |
Basic |
Metrics
editThe billing
dataset collects Cloud Billing Reports information from Google Cloud BigQuery daily cost detail table. BigQuery is a fully-managed, serverless data warehouse. Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify. Then you can access your Cloud Billing data from BigQuery for detailed analysis.
Please see export cloud billing data to BigQuery for more details on how to export billing data.
In BigQuery dataset, detailed Google Cloud daily cost data is loaded into a data table named gcp_billing_export_v1_<BILLING_ACCOUNT_ID>
. There is a defined schema for Google Cloud daily cost data that is exported to BigQuery. Please see daily cost detail data schema for more details.
For standard usage cost data, set the table pattern format to gcp_billing_export_v1
. This table pattern is set as the default when no other is specified.
For detailed usage cost data, set the table pattern to gcp_billing_export_resource_v1
. Detailed tables include the standard fields and additional fields, such as effective_price
, enabling a more granular view of expenses.
Requirements
editYou need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Before using any GCP integration you will need:
- GCP Credentials to connect with your GCP account.
- GCP Permissions to make sure the service account you’re using to connect has permission to share the relevant data.
To collect GCP Billing metrics, the following permissions are required to access the necessary data:
-
roles/bigquery.dataViewer
-
roles/bigquery.jobUser
-
roles/billing.viewer
Configuration Parameters
editdataset_id
editThe dataset_id
is the unique identifier of your BigQuery dataset where your billing data is stored. You can find this ID in your Google Cloud Console under the BigQuery section.
table_pattern
editThe table_pattern
parameter allows you to specify which tables to retrieve from the specified dataset. This can be set to either gcp_billing_export_v1
for standard usage cost data or gcp_billing_export_resource_v1
for detailed usage cost data.
cost_type
editThe cost_type
parameter enables you to filter the cost data based on specific cost categories. You can select one of the following options:
-
regular
: This cost type includes all the regular costs associated with your usage of GCP services. This does not include any taxes, adjustments, or rounding errors. -
tax
: This cost type includes all the taxes associated with your usage of GCP services. This does not include the regular costs, adjustments, or rounding errors. -
adjustment
: This cost type includes any adjustments made to your billing data. Adjustments can include credits, discounts, refunds, or any other modifications to the original costs. -
rounding_error
: This cost type includes any rounding errors that occurred when calculating your costs. These are typically very small amounts and are used to reconcile any discrepancies due to rounding.
Example Configuration
editHere’s an example of what your configuration might look like:
dataset_id: "my_billing_dataset" table_pattern: "gcp_billing_export_resource_v1" project_id: "my_project" cost_type: "regular"
In this example, the Agent will pull data from all tables within the my_billing_dataset
dataset that start with the pattern gcp_billing_export_resource_v1
.
Sample Event
editExample
An example event for billing
looks as following:
{ "@timestamp": "2017-10-12T08:05:34.853Z", "cloud": { "account": { "id": "01475F-5B1080-1137E7" }, "project": { "id": "elastic-bi", "name": "elastic-containerlib-prod" }, "provider": "gcp" }, "event": { "dataset": "gcp.billing", "duration": 115000, "module": "gcp" }, "gcp": { "billing": { "billing_account_id": "01475F-5B1080-1137E7", "cost_type": "regular", "invoice_month": "202106", "project_id": "containerlib-prod-12763", "project_name": "elastic-containerlib-prod", "total": 4717.170681, "sku_id": "0D56-2F80-52A5", "service_id": "6F81-5844-456A", "sku_description": "Network Inter Region Ingress from Jakarta to Americas", "service_description": "Compute Engine", "effective_price": 0.00292353, "tags": [ { "key": "stage", "value": "prod" }, { "key": "size", "value": "standard" } ] } }, "metricset": { "name": "billing", "period": 10000 }, "service": { "type": "gcp" } }
Exported fields
editECS Field Reference
Please refer to the following document for detailed information on ECS fields.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp |
Event timestamp. |
date |
cloud.image.id |
Image ID for the cloud instance. |
keyword |
data_stream.dataset |
Data stream dataset. |
constant_keyword |
data_stream.namespace |
Data stream namespace. |
constant_keyword |
data_stream.type |
Data stream type. |
constant_keyword |
event.dataset |
Event dataset |
constant_keyword |
event.module |
Event module |
constant_keyword |
gcp.billing.billing_account_id |
Project Billing Account ID. |
keyword |
gcp.billing.cost_type |
Cost types include regular, tax, adjustment, and rounding_error. |
keyword |
gcp.billing.effective_price |
The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it’s the list price. |
float |
gcp.billing.invoice_month |
Billing report month. |
keyword |
gcp.billing.project_id |
Project ID of the billing report belongs to. |
keyword |
gcp.billing.project_name |
Project Name of the billing report belongs to. |
keyword |
gcp.billing.service_description |
The Google Cloud service that reported the Cloud Billing data. |
keyword |
gcp.billing.service_id |
The ID of the service that the usage is associated with. |
keyword |
gcp.billing.sku_description |
A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US. |
keyword |
gcp.billing.sku_id |
The ID of the resource used by the service. |
keyword |
gcp.billing.tags |
A collection of key-value pairs that provide additional metadata. |
nested |
gcp.billing.tags.key |
keyword |
|
gcp.billing.tags.value |
keyword |
|
gcp.billing.total |
Total billing amount. |
float |
host.containerized |
If the host is a container. |
boolean |
host.os.build |
OS build information. |
keyword |
host.os.codename |
OS codename, if any. |
keyword |
Changelog
editChangelog
Version | Details | Kibana version(s) |
---|---|---|
2.38.0 |
Enhancement (View pull request) Bug fix (View pull request) |
8.13.0 or higher |
2.37.2 |
Bug fix (View pull request) |
8.13.0 or higher |
2.37.1 |
Enhancement (View pull request) |
8.13.0 or higher |
2.37.0 |
Enhancement (View pull request) |
8.13.0 or higher |
2.36.0 |
Enhancement (View pull request) |
8.13.0 or higher |
2.35.0 |
Enhancement (View pull request) |
8.13.0 or higher |
2.34.1 |
Bug fix (View pull request) |
8.12.0 or higher |
2.34.0 |
Enhancement (View pull request) |
8.12.0 or higher |
2.33.2 |
Enhancement (View pull request) |
8.12.0 or higher |
2.33.1 |
Enhancement (View pull request) |
8.12.0 or higher |
2.33.0 |
Enhancement (View pull request) |
8.12.0 or higher |
2.32.1 |
Enhancement (View pull request) |
8.12.0 or higher |
2.32.0 |
Enhancement (View pull request) |
8.12.0 or higher |
2.31.2 |
Bug fix (View pull request) |
8.7.1 or higher |
2.31.1 |
Enhancement (View pull request) |
8.7.1 or higher |
2.31.0 |
Enhancement (View pull request) |
8.7.1 or higher |
2.30.1 |
Bug fix (View pull request) |
8.7.1 or higher |
2.30.0 |
Enhancement (View pull request) Enhancement (View pull request) Bug fix (View pull request) Bug fix (View pull request) |
8.7.1 or higher |
2.29.1 |
Bug fix (View pull request) |
8.7.1 or higher |
2.29.0 |
Bug fix (View pull request) |
8.7.1 or higher |
2.28.5 |
Enhancement (View pull request) |
8.7.1 or higher |
2.28.4 |
Enhancement (View pull request) |
8.7.1 or higher |
2.28.3 |
Enhancement (View pull request) |
8.7.1 or higher |
2.28.2 |
Enhancement (View pull request) |
8.7.1 or higher |
2.28.1 |
Enhancement (View pull request) |
8.7.1 or higher |
2.28.0 |
Enhancement (View pull request) |
8.7.1 or higher |
2.27.0 |
Enhancement (View pull request) |
8.7.1 or higher |
2.26.0 |
Bug fix (View pull request) |
8.7.1 or higher |
2.25.1 |
Bug fix (View pull request) |
8.7.1 or higher |
2.25.0 |
Enhancement (View pull request) |
8.7.1 or higher |
2.24.0 |
Enhancement (View pull request) |
8.7.1 or higher |
2.23.0 |
Enhancement (View pull request) |
8.7.1 or higher |
2.22.1 |
Enhancement (View pull request) |
8.6.0 or higher |
2.22.0 |
Enhancement (View pull request) |
8.6.0 or higher |
2.21.0 |
Enhancement (View pull request) |
8.6.0 or higher |
2.20.1 |
Bug fix (View pull request) |
8.6.0 or higher |
2.20.0 |
Enhancement (View pull request) |
8.6.0 or higher |
2.19.1 |
Enhancement (View pull request) |
8.6.0 or higher |
2.19.0 |
Enhancement (View pull request) |
8.6.0 or higher |
2.18.0 |
Enhancement (View pull request) |
8.6.0 or higher |
2.17.2 |
Bug fix (View pull request) |
8.6.0 or higher |
2.17.1 |
Enhancement (View pull request) |
8.6.0 or higher |
2.17.0 |
Enhancement (View pull request) Enhancement (View pull request) Enhancement (View pull request) Enhancement (View pull request) |
8.6.0 or higher |
2.16.2 |
Bug fix (View pull request) |
8.5.0 or higher |
2.16.1 |
Bug fix (View pull request) |
8.5.0 or higher |
2.16.0 |
Enhancement (View pull request) |
8.5.0 or higher |
2.15.2 |
Enhancement (View pull request) |
8.5.0 or higher |
2.15.1 |
Enhancement (View pull request) |
8.5.0 or higher |
2.15.0 |
Enhancement (View pull request) Enhancement (View pull request) |
8.5.0 or higher |
2.14.0 |
Enhancement (View pull request) |
8.3.0 or higher |
2.13.0 |
Enhancement (View pull request) |
8.3.0 or higher |
2.12.1 |
Bug fix (View pull request) |
7.17.6 or higher |
2.12.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.12 |
Bug fix (View pull request) |
7.17.6 or higher |
2.11.11 |
Bug fix (View pull request) |
7.17.6 or higher |
2.11.10 |
Enhancement (View pull request) Enhancement (View pull request) Enhancement (View pull request) Enhancement (View pull request) Enhancement (View pull request) Enhancement (View pull request) |
7.17.6 or higher |
2.11.10-beta.6 |
Enhancement (View pull request) |
— |
2.11.10-beta.5 |
Enhancement (View pull request) |
— |
2.11.10-beta.4 |
Enhancement (View pull request) |
— |
2.11.10-beta.3 |
Enhancement (View pull request) |
— |
2.11.10-beta.2 |
Enhancement (View pull request) |
— |
2.11.10-beta.1 |
Enhancement (View pull request) |
— |
2.11.9 |
Bug fix (View pull request) |
7.17.6 or higher |
2.11.8 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.7 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.6 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.5 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.4 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.3 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.2 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.1 |
Enhancement (View pull request) |
7.17.6 or higher |
2.11.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.10.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.9.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.8.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.7.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.6.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.5.0 |
Enhancement (View pull request) Bug fix (View pull request) Bug fix (View pull request) Bug fix (View pull request) Enhancement (View pull request) |
7.17.6 or higher |
2.4.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.3.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.2.1 |
Enhancement (View pull request) |
7.17.6 or higher |
2.2.0 |
Enhancement (View pull request) |
7.17.6 or higher |
2.1.0 |
Enhancement (foobar[View pull request]) |
7.17.6 or higher |
2.0.0 |
Breaking change (View pull request) This change involves Users with GCP integration enabled will need to input values for these variables again when upgrading the policies to this version. Enhancement (View pull request) Enhancement (View pull request) Enhancement (View pull request) |
8.3.0 or higher |
1.10.0 |
Enhancement (View pull request) |
7.17.0 or higher |
1.9.2 |
Bug fix (View pull request) |
7.17.0 or higher |
1.9.1 |
Enhancement (View pull request) |
7.17.0 or higher |
1.9.0 |
Enhancement (View pull request) |
7.17.0 or higher |
1.8.0 |
Enhancement (View pull request) |
7.17.0 or higher |
1.7.0 |
Enhancement (View pull request) Bug fix (View pull request) |
— |
1.6.1 |
Enhancement (View pull request) |
7.16.3 or higher |
1.6.0 |
Enhancement (View pull request) |
— |
1.5.1 |
Enhancement (View pull request) |
7.16.3 or higher |
1.5.0 |
Enhancement (View pull request) |
7.16.3 or higher |
1.4.2 |
Bug fix (View pull request) |
7.16.3 or higher |
1.4.1 |
Bug fix (View pull request) |
7.16.3 or higher |
1.4.0 |
Enhancement (View pull request) |
— |
1.3.1 |
Bug fix (View pull request) |
7.15.0 or higher |
1.3.0 |
Enhancement (View pull request) |
7.15.0 or higher |
1.2.2 |
Bug fix (View pull request) |
7.15.0 or higher |
1.2.1 |
Bug fix (View pull request) |
— |
1.2.0 |
Enhancement (View pull request) |
7.15.0 or higher |
1.1.2 |
Enhancement (View pull request) |
7.15.0 or higher |
1.1.1 |
Bug fix (View pull request) |
— |
1.1.0 |
Enhancement (View pull request) |
7.15.0 or higher |
1.0.0 |
Enhancement (View pull request) Enhancement (View pull request) |
— |
0.3.3 |
Enhancement (View pull request) |
— |
0.3.2 |
Enhancement (View pull request) |
— |
0.3.1 |
Enhancement (View pull request) |
— |
0.3.0 |
Enhancement (View pull request) |
— |
0.2.0 |
Enhancement (View pull request) |
— |
0.1.0 |
Enhancement (View pull request) |
— |
0.0.2 |
Enhancement (View pull request) |
— |
0.0.1 |
Enhancement (View pull request) |
— |