Configure EDOT Cloud Forwarder for AWS
Before deploying EDOT Cloud Forwarder for AWS, configure the CloudFormation template parameters based on your specific requirements. The template uses the following settings to deploy and configure the EDOT Collector Lambda function.
The CloudFormation templates are hosted in a public Amazon S3 bucket and are accessible through HTTPS URL. You can reference these templates directly during deployment or download them for local use.
| Log type | Log source | CloudFormation template |
|---|---|---|
| VPC | S3 | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml |
| ELB | S3 | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml |
| CloudTrail | S3 | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml |
| VPC | CloudWatch | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/cloudwatch_logs-cloudformation.yaml |
| CloudTrail | CloudWatch | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/cloudwatch_logs-cloudformation.yaml |
For specific versions, edit latest in the URL to the required version in the format vX.Y.Z.
These are the required settings:
| Setting | Description |
|---|---|
stack-name |
Name of the CloudFormation stack, for example, vpc-edot-cfDo not use the same name for different stacks. |
OTLPEndpoint |
The OTLP endpoint URL used for data ingestion, obtained from Elastic Cloud Serverless. |
ElasticApiKey |
API key for authentication with Elastic, obtained from Elastic Cloud Serverless. |
Set the following settings based on the log source:
For logs sourced from S3, use the following settings:
| Setting | Description |
|---|---|
EdotCloudForwarderS3LogsType |
The encoding format for logs in the S3 bucket. Supported options: - vpcflow: VPC Flow Logs- elbaccess: ELB Access logs- cloudtrail: CloudTrail Logs- waf: WAF Logs |
SourceS3BucketARN |
Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the edot-cloud-forwarder Lambda function automatically. |
For logs sourced from CloudWatch, use the following settings:
| Setting | Description |
|---|---|
SourceCloudWatchLogGroupARN |
Amazon Resource Name (ARN) of the CloudWatch Log Group where the subscription filter will be created. The ARN must include the trailing :*. |
EdotCloudForwarderCWLogType |
The type of logs in the CloudWatch Log Group. Supported options: - vpcflow: VPC Flow Logs- cloudtrail: CloudTrail Logs |
The log group must already exist in your AWS account and region. If the ARN points to a non-existent log group, stack deployment or updates will fail.
These are optional settings you can set in the CloudFormation template:
| Setting | Description |
|---|---|
EdotCloudForwarderConcurrentExecutions |
Sets the maximum number of reserved concurrent executions for the Lambda function. Default value is 5 for S3 sources and 20 for CloudWatch sources. CloudWatch sources require significantly higher concurrency than S3 for the same data rate (see CloudWatch Logs sizing). If you notice Lambda throttling, increase concurrent executions. Make sure this value doesn't exceed your AWS account's concurrency limit. |
EdotCloudForwarderMemorySize |
Sets the allocated memory for the Lambda function, measured in megabytes. The default value is 512 MB. Minimum value is 128 MB. Maximum value is 10240 MB. For S3 sources, 512 MB is recommended. For CloudWatch sources, 256 MB is sufficient — the smaller payload sizes require less memory. More memory increases Lambda CPU allocation, increasing processing speed of events. |
EdotCloudForwarderTimeout |
Maximum execution time for the Lambda function, measured in seconds. The default is set to 900 seconds or 15 minutes. Accepts values from 1 second to 900 seconds. |
EdotCloudForwarderVersion |
Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example 1.0.0. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. |
EdotCloudForwarderExporterMaxQueueSize |
Sets the internal OTLP exporter queue size, measured in bytes. The default value is 50000000 (50 MB). This parameter should be used only in exceptional edge cases that require manual tuning of the export queue. |
EdotCloudForwarderLogLevel |
Sets the log level for the EDOT Cloud Forwarder collector. Accepted values: debug, info, warn, error. Default value is info. Set to debug when troubleshooting to produce detailed logs in the Lambda CloudWatch log group. |
The default values provided have been determined through extensive load testing across different log types and data volumes. For most use cases, these defaults provide a good balance between cost and performance.
Adjust these parameters only if you notice performance issues such as Lambda timeouts, throttling, high memory usage or dropped data. If you need assistance tuning these parameters for your specific workload, refer to Contact support.
Use the following sizing suggestions to select appropriate reserved concurrency (EdotCloudForwarderConcurrentExecutions) and Lambda memory (EdotCloudForwarderMemorySize) based on your expected traffic volumes. This helps maximize performance and prevent Lambda throttling at high log volumes.
Monitor CloudWatch metrics for Lambda throttling and concurrent executions, as well as CloudWatch Logs for execution duration per Lambda invocation.
| Throughput | Log rate | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|---|
| < 5 MB/s | < 50,000 logs/s | 5 | 512 MB | Default configuration |
| 5 - 10 MB/s | 50,000 - 100,000 logs/s | 10 | 512 MB | Increase concurrency |
| > 10 MB/s | > 100,000 logs/s | > 10 | 512 MB | First increase concurrency and then increase memory as needed |
| Throughput | Log rate | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|---|
| < 10 MB/s | < 25,000 events/s | 5 | 512 MB | Default configuration |
| 10 - 40 MB/s | 25,000 - 100,000 events/s | 20 | 512 MB | Increase concurrency |
| > 40 MB/s | > 100,000 events/s | > 20 | 512 MB | First increase concurrency and then increase memory as needed |
ELB logs might produce files with gigabytes of data. The default configurations are confirmed to work up to 3GB log files, which translates roughly to 23,000 requests per second per ELB. If you expect higher request volumes, increase Lambda memory allocation.
| Throughput | Log rate | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|---|
| ≤ 10 MB/s | < 10,000 events/s | 5 | 512 MB | Default configuration |
| 10 - 30 MB/s | 10,000 - 30,000 events/s | 10 | 512 MB | Increase concurrency |
| > 30 MB/s | > 30,000 events/s | 15+ | 512 MB | Scale concurrency as needed |
CloudWatch delivers logs differently from S3: instead of large files, it sends many small batches that each trigger a separate Lambda invocation. This results in significantly higher concurrency requirements for the same data rate.
| Throughput | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|
| < 1 MB/s | 20 | 256 MB | Default configuration |
| 1 - 8 MB/s | 100 | 256 MB | Increase concurrency |
| > 8 MB/s | 200 | 256 MB | Scale concurrency as needed |
These recommendations apply to both VPC Flow Logs and CloudTrail logs delivered through CloudWatch.
- Deployment methods: Deploy using AWS CLI, AWS Console, or AWS Serverless Application Repository.
- Troubleshooting: Diagnose and resolve issues with log forwarding.