Loading

EDOT Cloud Forwarder for AWS

EDOT Cloud Forwarder (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Cloud Managed OTLP Endpoint.

EDOT Cloud Forwarder for AWS supports the following log sources:

AWS log type Telemetry description Via S3 Via CloudWatch
VPC Flow Logs VPC Flow Logs to capture information about IP traffic.
ELB Access Logs Access logs for your Application Load Balancer. N/A
CloudTrail Logs CloudTrail Logs to record account activity.
WAF Logs WAF Logs to capture web request details for security analysis. Not yet available

To collect logs using EDOT Cloud Forwarder for AWS, you need the following:

Via S3

To collect VPC Flow logs from S3, you need:

  • A Virtual Private Cloud (VPC)
  • An S3 bucket for storing flow logs
  • A flow log configured with the S3 bucket as the destination
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via CloudWatch

To collect VPC Flow logs from CloudWatch, you need:

  • A Virtual Private Cloud (VPC) with flow logs delivered to a CloudWatch Log Group
  • The Log Group ARN (must include the trailing :*)
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect Elastic Load Balancer (ELB) Access logs, you need:

  • An ELB of any type (ALB, NLB, CLB)
  • An S3 bucket to store the access logs
  • Access logging enabled, with the bucket as the destination
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect CloudTrail logs from S3, you need:

  • A trail that delivers account events as log files to an Amazon S3 bucket
  • An S3 bucket to store the trail logs
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via CloudWatch

To collect CloudTrail logs from CloudWatch, you need:

  • CloudTrail configured to deliver logs to a CloudWatch Log Group
  • The Log Group ARN (must include the trailing :*)
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect AWS WAF logs, you need:

  • AWS WAF with logging enabled to an S3 bucket
  • An S3 bucket to store the WAF logs
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

To retrieve your Elastic Cloud Managed OTLP Endpoint endpoint address and API key, follow these steps:

  1. In Elastic Cloud, create an Observability project or open an existing one.
  2. Go to Add data, select Applications and then select OpenTelemetry.
  3. Copy the endpoint and authentication headers values.

Alternatively, you can retrieve the endpoint from the Manage project page and create an API key manually from the API keys page.

  1. Log in to the Elastic Cloud Console.
  2. Find your deployment on the home page or on the Hosted deployments page, and then select Manage.
  3. In the Application endpoints, cluster and component IDs section, select Managed OTLP.
  4. Copy the public endpoint value.
Important

Trim the API key from Authorization=ApiKey MYKEYVALUE... to just MYKEYVALUE... before using it as the argument to the ElasticAPIKey parameter.

Deploy EDOT Cloud Forwarder for AWS with one click using the AWS CloudFormation console:

Launch Stack

After clicking the button:

  1. Configure the required parameters:

    Parameter Description
    Stack name Name of the CloudFormation stack, for example vpc-edot-cf.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticApiKey API key for authentication with Elastic.
    SourceS3BucketARN ARN of the S3 bucket where your logs are stored.
    EdotCloudForwarderS3LogsType The log type: vpcflow, elbaccess, cloudtrail, or waf.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

Launch Stack

After clicking the button:

  1. Configure the required parameters:

    Parameter Description
    Stack name Name of the CloudFormation stack, for example cw-vpc-edot-cf.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticApiKey API key for authentication with Elastic.
    SourceCloudWatchLogGroupARN ARN of the CloudWatch Log Group where your logs are delivered, including the trailing :*.
    EdotCloudForwarderCWLogType The log type: vpcflow or cloudtrail.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

Tip

The CloudFormation stack deployment region must match the region of your log source (S3 bucket or CloudWatch Log Group).

Before deploying EDOT Cloud Forwarder for AWS, consider the following:

  • Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time.
  • S3 sources: Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.
  • CloudWatch sources: Each CloudFormation stack subscribes to a single CloudWatch Log Group. Deploy a separate stack for each Log Group you want to forward.
  • S3 and CloudWatch stacks are independent — you can deploy both to collect the same log type from different sources.

For log types that support both S3 and CloudWatch (VPC Flow Logs and CloudTrail), the choice depends on your delivery requirements and existing setup:

  • Use S3 if your logs are already delivered to S3, or if periodic batch delivery (every ~5 minutes) is acceptable for your use case. S3 is the most cost-effective option for log delivery.
  • Use CloudWatch if you need near-real-time log delivery, or if your logs are already published to a CloudWatch Log Group. Note that CloudWatch log delivery costs are significantly higher than S3.

Both sources produce identical data in Elastic: the same datastreams, field mappings, and dashboards apply regardless of the source.

Logs collected by EDOT Cloud Forwarder for AWS are stored in Elasticsearch datastreams in OpenTelemetry native format. The following table shows which datastreams are used for each log type:

AWS log type Log source Datastream dataset Description
VPC Flow Logs S3 or CloudWatch aws.vpcflow.otel VPC Flow Log records
ELB Access Logs S3 aws.elbaccess.otel ELB Access Log records (ALB, NLB, CLB)
CloudTrail Logs S3 or CloudWatch aws.cloudtrail.otel CloudTrail account activity records
WAF Logs S3 aws.waf.otel AWS WAF web request log records

Both S3 and CloudWatch sources produce the same datastream format for the same log type.

The logs are produced in OpenTelemetry native format. For detailed information about the field mappings and structure of each log type, refer to the following documentation:

After EDOT Cloud Forwarder for AWS is successfully running and forwarding logs to Elastic Observability, install the Kibana integrations to visualize your data with out-of-the-box dashboards and visualizations.

To set up data visualization in Kibana:

  1. Log into your Elastic Cloud deployment and open Kibana

  2. Go to Management → Integrations in the Kibana navigation menu.

  3. Search for the appropriate integration based on your log type and install it:

    AWS log type Integration name Description
    ELB Access Logs AWS ELB OpenTelemetry Assets Dashboards and visualizations for Elastic Load Balancer logs
    VPC Flow Logs AWS VPC Flow Logs OpenTelemetry Assets Dashboards and visualizations for VPC flow log data
    CloudTrail Logs AWS CloudTrail Logs OpenTelemetry Assets Dashboards and visualizations for CloudTrail log data
    WAF Logs AWS WAF Logs OpenTelemetry Assets Dashboards and visualizations for WAF log data

  4. Once installed, navigate to Dashboard to view the pre-built dashboards for your AWS log data.

EDOT Cloud Forwarder for AWS has the following limitations:

Limitation Description
VPC/PrivateLink not supported EDOT Cloud Forwarder cannot be deployed inside a VPC or use AWS PrivateLink endpoints. The Lambda function requires public internet access to forward data to the OTLP endpoint.
Managed OTLP Input only EDOT Cloud Forwarder is tested exclusively with Elastic Cloud Managed OTLP Endpoint. Forwarding to a self-deployed EDOT Collector Gateway is not tested and forwarding to APM Server is not supported.
Single log type per S3 bucket Each S3 bucket can only contain one log type. Mixed log formats in the same bucket are not supported yet.
Single Log Group per CloudWatch stack Each CloudWatch stack subscribes to one Log Group. Deploy a separate stack for each Log Group you want to forward.
  • Configure the template: Learn about all configuration options, including optional settings and sizing recommendations.
  • Deployment methods: Explore alternative deployment methods using AWS CLI or AWS Serverless Application Repository.
  • Troubleshooting: Diagnose and resolve issues with log forwarding.