Elastic Cloud with AWS FireLens: Accelerate time to insight with agentless data ingestion

By

Akshay Saraswat,

Udayasimha Theepireddy (Uday)

blog-thumb-cloud-aws.png

Customers can now directly ship container logs and events to Elastic Cloud with Amazon Web Services (AWS) FireLens. AWS FireLens is a container log router for Amazon Elastic Container Service (Amazon ECS) launch types: Amazon Elastic Compute Cloud (Amazon EC2) and AWS Fargate. This integration enables DevOps and DevSecOps experts to streamline AWS log management from shipping and cost-efficient log retention, to monitoring and analyzing threats.

In this blog, we will cover how to get started with agentless data ingestion to Elastic Cloud using AWS FireLens.

Streamline log and event shipping

Traditionally, shipping container logs and events to different destinations has been a lengthy and challenging process. Say goodbye to time-consuming tasks like sending logs to Cloudwatch, maintaining Fluent Bit sidecar log aggregators, manually installing extra software, or writing additional code.You can now quickly configure the task definition to send application logs running on ECS to Elastic Cloud.

Simplify searching, analyzing, and retaining logs

Once you stream data from FireLens and store it in Elasticsearch, you can search and visualize it in Kibana in minutes. You can also ship logs, metrics, and traces from your other servers, virtual machines, and containers to Elastic and analyze them through a single pane of glass.

Get deep application visibility and reduce the time spent on root cause analysis with Elastic Observability. Quickly and easily analyze code–level traces, logs, and metrics, map complex service dependencies, and surface “unknown unknowns” with machine learning–based anomaly detection and automated correlations.

Drive mean time to detection towards zero with Elastic Security. Expand the usage of the same data for extended protection and response (XDR). Bring together SIEM and endpoint security data to prevent, detect, and respond to the latest cybersecurity threats.

As your data grows, Elastic’s autoscaling and index lifecycle management capabilities ensure no data is dropped while balancing performance and storage costs.

Getting started with Elastic and AWS FireLens

Let’s start with an example of using FireLens to forward container logs from Amazon ECS to Elastic Cloud. This reference architecture diagram is a relatively standard depiction of FireLens with a container in Amazon ECS that is forwarding container logs to Elastic Cloud.
forward container logs from Amazon ECS to Elastic Cloud

FireLens task configuration steps:

1. Configure a task definition. You can find an example definition from the Amazon ECS FireLens Examples on GitHub.

2. Update and replace the AWS Identity and Access Management (IAM) roles with your own task RoleArn,executionRoleArn IAM roles, Elastic Cloud_Auth and Cloud_ID. The updated task definition and readme to implement this solution can be found in the GitHub Repository. You can also view the task details below as an example.
{
   "family": "firelens-EC2-elastic",
   "taskRoleArn": "CHANGE ME",
   "executionRoleArn": "CHANGE ME",
   "containerDefinitions": [{
           "name": "log_router",
           "image": "amazon/aws-for-fluent-bit:latest",
           "essential": true,
           "firelensConfiguration": {
               "type": "fluentbit",
               "options": {
                   "enable-ecs-log-metadata": "true"
               }
           },
           "logConfiguration": {
               "logDriver": "awslogs",
               "options": {
                   "awslogs-create-group": "true",
                   "awslogs-group": "firelens-container",
                   "awslogs-region": "us-east-2",
                   "awslogs-stream-prefix": "firelens"
               }
           },
           "memoryReservation": 50
       },
       {
           "name": "nginx-test",
           "image": "nginx",
           "portMappings": [{
               "containerPort": 80
           }],
           "essential": true,
           "environment": [],
           "logConfiguration": {
               "logDriver": "awsfirelens",
               "secretOptions": [
                       {
                               "valueFrom": "CHANGE ME",
                               "name": "Cloud_Auth"
                       }
               ],
               "options": {
                   "Name": "es",
                   "Port": "9243",
                   "Tag_Key tags": "tags",
                   "Include_Tag_Key": "true",
                   "Cloud_ID": "CHANGE ME",
                   "Index": "elastic_firelens”,
                   "tls": "On",
                   "tls.verify": "Off"
               }
           },
           "memoryReservation": 100
       }
   ]
}

If you’re planning to ship Amazon ECS Fargate container logs, replace the first few lines with the configuration below.

{
    "family": "firelens-fargate-elastic",
    "taskRoleArn": "",
    "taskRoleArn": "CHANGE ME",
    "executionRoleArn": "CHANGE ME",
    "cpu": "512",
    "memory": "1024",
    "requiresCompatibilities": [
        "FARGATE"
    ],

Configuration steps:

1. Make sure ECS CLI and AWS CLI are installed, if not, first follow this installation guide from AWS.

2. Open a new file with name `enable-fluent-log-driver.sh` and store the info below to enable the Fluentd log driver in the ECS agent:

#!/bin/bash
echo "ECS_AVAILABLE_LOGGING_DRIVERS=[\"awslogs\",\"fluentd\"]" >> 
/etc/ecs/ecs.config
3. Create an EC2 or Fargate cluster on ECS with the following commands
ecs-cli up \
--size 2 \
--instance-type t2.medium \
--extra-user-data enable-fluent-log-driver.sh \
--keypair <changeme> \
--capability-iam \
--region=us-east-2 \
--ecs-profile=<changeme> \
--cluster-config <changeme>

4. Save task definition into a local file `task_definition.json` and register it using AWS CLI:

aws ecs register-task-definition --cli-input-json file://task_definition.json

5. Create ECS service:

aws ecs create-service --cluster cluster-name --service-name demo-service --task-definition firelens-example-cloudwatch --desired-count 1 --launch-type 
"EC2"

6. When service starts running, you should be able to see logs flowing into Elasticsearch through Kibana. The image below depicts a query run in Elastic that can be visualized in Kibana.

When service starts running, you should be able to see logs flowing into Elasticsearch through Kibana. The image below depicts a query run in Elastic that can be visualized in Kibana.

In Kibana, you can also visualize the data that you have in your Elastic indices. You can then build dashboards that display related visualizations based on your queries that are run on Elastic. By using Elastic aggregations to extract and process the data, you can create charts and graphs that show the spikes, dips, and trends across your data. The image below shows an example of a Kibana dashboard.

The image below shows an example of a Kibana dashboard.

The Elastic and AWS partnership includes technical integrations like FireLens to simplify data ingestion, and Elastic offers a single UI with inherent security and compliance control for DevOps and DevSecOps experts. Elastic's ready-to-use AWS integrations and dashboards allow you to unify visibility across your AWS and on-premises environments, enabling better insights into the performance and overall health of your infrastructure, applications, and business.

Wrapping up

Are you ready to streamline data ingestion and manage all your container logs and events from a single pane of glass? Elastic is constantly iterating to provide you with a frictionless experience, enabling you to run where you want and use what you want — and this streamlined integration with AWS is just one example of that. Elastic Cloud extends the value of the Elastic Stack, allowing you to do more, and faster, making it the best way to experience our platform. For more information on other AWS integrations, visit the Elastic integrations library.

Or, start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial