Custom AWS Log Integration
editCustom AWS Log Integration
editVersion |
1.5.1 (View all) |
Compatible Kibana version(s) |
8.16.0 or higher |
Supported Serverless project types |
Security |
Subscription level |
Basic |
Level of support |
Elastic |
The custom AWS input integration offers users two ways to collect logs from AWS: from an S3 bucket (with or without SQS notification) and from CloudWatch. Custom ingest pipelines may be added by adding the name to the pipeline configuration option, creating custom ingest pipelines can be done either through the API or the Ingest Node Pipeline UI.
Collecting logs from S3 bucket
editWhen collecting logs from S3 bucket is enabled, users can retrieve logs from S3 objects that are pointed to by S3 notification events read from an SQS queue or directly polling list of S3 objects in an S3 bucket.
The use of SQS notification is preferred: polling list of S3 objects is expensive in terms of performance and costs and should be preferably used only when no SQS notification can be attached to the S3 buckets. This input integration also supports S3 notification from SNS to SQS.
You can enable SQS notification method by setting queue_url
configuration value. You can enable S3 bucket list polling method by setting bucket_arn
configuration value
and number_of_workers
value. Both queue_url
and bucket_arn
cannot be set
at the same time and at least one of the two value must be set.
To access SQS and S3, these specific AWS permissions are required.
Collecting logs from CloudWatch
editWhen collecting logs from CloudWatch is enabled, users can retrieve logs from
all log streams in a specific log group. filterLogEvents
AWS API is used to
list log events from the specified log group. Amazon CloudWatch Logs can be used
to store log files from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail,
Route53, and other sources.
To access aws-cloudwatch, these specific AWS permissions are required.
Changelog
editChangelog
Version | Details | Kibana version(s) |
---|---|---|
1.5.1 |
Enhancement (View pull request) |
8.16.0 or higher |
1.5.0 |
Enhancement (View pull request) |
8.16.0 or higher |
1.4.2 |
Bug fix (View pull request) |
8.12.0 or higher |
1.4.1 |
Bug fix (View pull request) |
8.12.0 or higher |
1.4.0 |
Enhancement (View pull request) |
8.12.0 or higher |
1.3.1 |
Bug fix (View pull request) |
8.12.0 or higher |
1.3.0 |
Enhancement (View pull request) |
8.12.0 or higher |
1.2.0 |
Enhancement (View pull request) |
8.12.0 or higher |
1.1.0 |
Enhancement (View pull request) |
8.12.0 or higher |
1.0.0 |
Enhancement (View pull request) |
8.0.0 or higher |
0.6.0 |
Enhancement (View pull request) |
— |
0.5.1 |
Enhancement (View pull request) |
— |
0.5.0 |
Enhancement (View pull request) |
— |
0.4.0 |
Enhancement (View pull request) |
— |
0.3.3 |
Enhancement (View pull request) |
— |
0.3.2 |
Enhancement (View pull request) |
— |
0.3.1 |
Bug fix (View pull request) |
— |
0.3.0 |
Enhancement (View pull request) |
— |
0.2.5 |
Bug fix (View pull request) |
— |
0.2.4 |
Bug fix (View pull request) |
— |
0.2.3 |
Bug fix (View pull request) |
— |
0.2.2 |
Bug fix (View pull request) |
— |
0.2.1 |
Bug fix (View pull request) |
— |
0.2.0 |
Enhancement (View pull request) |
— |
0.1.0 |
Enhancement (View pull request) |
— |