Elastic Cloud with AWS FireLens: Accelerate time to insight with agentless data ingestion

blog-thumb-cloud-aws.png

Customers can now directly ship container logs and events to Elastic Cloud with Amazon Web Services (AWS) FireLens. AWS FireLens is a container log router for Amazon Elastic Container Service (Amazon ECS) launch types: Amazon Elastic Compute Cloud (Amazon EC2) and AWS Fargate. This integration enables DevOps and DevSecOps experts to streamline AWS log management from shipping and cost-efficient log retention, to monitoring and analyzing threats.

In this blog, we will cover how to get started with agentless data ingestion to Elastic Cloud using AWS FireLens.

Streamline log and event shipping

Traditionally, shipping container logs and events to different destinations has been a lengthy and challenging process. Say goodbye to time-consuming tasks like sending logs to Cloudwatch, maintaining Fluent Bit sidecar log aggregators, manually installing extra software, or writing additional code.You can now quickly configure the task definition to send application logs running on ECS to Elastic Cloud.

Simplify searching, analyzing, and retaining logs

Once you stream data from FireLens and store it in Elasticsearch, you can search and visualize it in Kibana in minutes. You can also ship logs, metrics, and traces from your other servers, virtual machines, and containers to Elastic and analyze them through a single pane of glass.

Get deep application visibility and reduce the time spent on root cause analysis with Elastic Observability. Quickly and easily analyze code–level traces, logs, and metrics, map complex service dependencies, and surface “unknown unknowns” with machine learning–based anomaly detection and automated correlations.

Drive mean time to detection towards zero with Elastic Security. Expand the usage of the same data for extended protection and response (XDR). Bring together SIEM and endpoint security data to prevent, detect, and respond to the latest cybersecurity threats.

As your data grows, Elastic’s autoscaling and index lifecycle management capabilities ensure no data is dropped while balancing performance and storage costs.

Getting started with Elastic and AWS Firelens

Let’s start with an example of using FireLens to forward container logs from Amazon ECS to Elastic Cloud. This reference architecture diagram is a relatively standard depiction of FireLens with a container in Amazon ECS that is forwarding container logs to Elastic Cloud.
forward container logs from Amazon ECS to Elastic Cloud

FireLens task configuration steps:

1. Configure a task definition. You can find an example definition from the Amazon ECS FireLens Examples on GitHub.

2. Update and replace the AWS Identity and Access Management (IAM) roles with your own task RoleArn,executionRoleArn IAM roles, Elastic Cloud_Auth and Cloud_ID. The updated task definition and readme to implement this solution can be found in the GitHub Repository. You can also view the task details below as an example.
{
   "family": "firelens-EC2-elastic",
   "taskRoleArn": "CHANGE ME",
   "executionRoleArn": "CHANGE ME",
   "containerDefinitions": [{
           "name": "log_router",
           "image": "amazon/aws-for-fluent-bit:latest",
           "essential": true,
           "firelensConfiguration": {
               "type": "fluentbit",
               "options": {
                   "enable-ecs-log-metadata": "true"
               }
           },
           "logConfiguration": {
               "logDriver": "awslogs",
               "options": {
                   "awslogs-create-group": "true",
                   "awslogs-group": "firelens-container",
                   "awslogs-region": "us-east-2",
                   "awslogs-stream-prefix": "firelens"
               }
           },
           "memoryReservation": 50
       },
       {
           "name": "nginx-test",
           "image": "nginx",
           "portMappings": [{
               "containerPort": 80
           }],
           "essential": true,
           "environment": [],
           "logConfiguration": {
               "logDriver": "awsfirelens",
               "secretOptions": [
                       {
                               "valueFrom": "CHANGE ME",
                               "name": "Cloud_Auth"
                       }
               ],
               "options": {
                   "Name": "es",
                   "Port": "9243",
                   "Tag_Key tags": "tags",
                   "Include_Tag_Key": "true",
                   "Cloud_ID": "CHANGE ME",
                   "Index": "elastic_firelens”,
                   "tls": "On",
                   "tls.verify": "Off"
               }
           },
           "memoryReservation": 100
       }
   ]
}Read more

If you’re planning to ship Amazon ECS Fargate container logs, replace the first few lines with the configuration below.

{
    "family": "firelens-fargate-elastic",
    "taskRoleArn": "",
    "taskRoleArn": "CHANGE ME",
    "executionRoleArn": "CHANGE ME",
    "cpu": "512",
    "memory": "1024",
    "requiresCompatibilities": [
        "FARGATE"
    ],

Configuration steps:

1. Make sure ECS CLI and AWS CLI are installed, if not, first follow this installation guide from AWS.

2. Open a new file with name `enable-fluent-log-driver.sh` and store the info below to enable the Fluentd log driver in the ECS agent:

#!/bin/bash
echo "ECS_AVAILABLE_LOGGING_DRIVERS=[\"awslogs\",\"fluentd\"]" >> 
/etc/ecs/ecs.config
3. Create an EC2 or Fargate cluster on ECS with the following commands
ecs-cli up \
--size 2 \
--instance-type t2.medium \
--extra-user-data enable-fluent-log-driver.sh \
--keypair <changeme> \
--capability-iam \
--region=us-east-2 \
--ecs-profile=<changeme> \
--cluster-config <changeme>

4. Save task definition into a local file `task_definition.json` and register it using AWS CLI:

We're hiring

Work for a global, distributed team where finding someone like you is just a Zoom meeting away. Flexible work with impact? Development opportunities from the start?