Custom GCS (Google Cloud Storage) Input

edit

Custom GCS (Google Cloud Storage) Input

edit

Version

2.1.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Elastic

Use the Google Cloud Storage input to read content from files stored in buckets that reside on your Google Cloud. The input can be configured to work with and without polling, though currently, if polling is disabled it will only perform a one-time passthrough, list the file contents and end the process. Polling is generally recommended for most cases even though it can get expensive with dealing with a very large number of files.

To mitigate errors and ensure a stable processing environment, this input employs the following features :

  1. When processing Google Cloud buckets, if suddenly there is any outage, the process will be able to resume post the last file it processed and for which it was successfully able to save the state.
  2. If any errors occur for certain files, they will be logged appropriately, but the rest of the files will continue to be processed normally.
  3. If any major error occurs that stops the main thread, the logs will be appropriately generated, describing said error.

Currently only JSON is supported with respect to object/file formats. We also support gzipped JSON objects/files. As for authentication types, we currently have support for JSON credential keys and credential files. If a download for a file/object fails or gets interrupted, the download is retried two times which equates to a maximum of three tries for a particular file/object. This is behavior currently not user-configurable.

The GCS credentials key file:

edit

This is a one-time download JSON key file that you get after adding a key to a GCP service account. If you are just starting out creating your GCS bucket, do the following:

  1. Make sure you have a service account available, if not follow the steps below:

    • Navigate to APIs & Services > Credentials
    • Click on Create credentials > Service account
  2. Once the service account is created, you can navigate to the Keys section and attach/generate your service account key.
  3. Make sure to download the JSON key file once prompted.
  4. Use this JSON key file either inline (JSON string object), or by specifying the path to the file on the host machine, where the agent is running.

A sample JSON Credentials file looks as follows:

{
  "type": "dummy_service_account",
  "project_id": "dummy-project",
  "private_key_id": "dummy-private-key-id",
  "private_key": "-----BEGIN PRIVATE KEY-----\nDummyPrivateKey\n-----END PRIVATE KEY-----\n",
  "client_email": "dummy-service-account@example.com",
  "client_id": "12345678901234567890",
  "auth_uri": "https://dummy-auth-uri.com",
  "token_uri": "https://dummy-token-uri.com",
  "auth_provider_x509_cert_url": "https://dummy-auth-provider-cert-url.com",
  "client_x509_cert_url": "https://dummy-client-cert-url.com",
  "universe_domain": "dummy-universe-domain.com"
}

NOTE:

  • When using the GCS integration, if you are using JSON Credentials inline, then you must specify the entire JSON object within single quotes i.e '{GCS_CREDS_JSON_OBJECT}'

Configuring The Input:

edit

Assuming you have GCS buckets already set up and the service account key available, please refer to the input documentation here for further details on specific parameters used by the integration.

ECS Field Mapping

edit

This integration includes the ECS Dynamic Template, all fields that follow the ECS Schema will get assigned the correct index field mapping and do not need to be added manually.

Ingest Pipelines

edit

Custom ingest pipelines may be added by adding the name to the pipeline configuration option, creating custom ingest pipelines can be done either through the API or the Ingest Node Pipeline UI.

Changelog

edit
Changelog
Version Details Kibana version(s)

2.1.0

Enhancement (View pull request)
ECS version updated to 8.11.0. Removed import_mappings. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

2.0.0

Enhancement (View pull request)
Convert Google Cloud Storage to input package type.

8.12.0 or higher

1.1.0

Enhancement (View pull request)
Set sensitive values as secret.

8.12.0 or higher

1.0.1

Enhancement (View pull request)
Changed owners

8.11.0 or higher

1.0.0

Enhancement (View pull request)
Made integration GA. Refactored integration by adding support for new parameters and updated documentation.

8.11.0 or higher

0.9.0

Enhancement (View pull request)
ECS version updated to 8.11.0.

0.8.0

Enhancement (View pull request)
ECS version updated to 8.10.0.

0.7.0

Enhancement (View pull request)
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added owner.type: elastic to package manifest.

0.6.0

Enhancement (View pull request)
Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

0.5.0

Enhancement (View pull request)
Update package to ECS 8.9.0.

0.4.0

Enhancement (View pull request)
Update package to ECS 8.8.0.

0.3.0

Enhancement (View pull request)
Update package-spec version to 2.7.0.

0.2.0

Enhancement (View pull request)
Update package to ECS 8.7.0.

0.1.0

Enhancement (View pull request)
Initial Release