Managing your applications on Amazon ECS EC2-based clusters with Elastic Observability

By

Bahubali Shetti

library-branding-elastic-observability-midnight-1680x980.png

In previous blogs, we explored how Elastic Observability can help you monitor various AWS services and analyze them effectively:

One of the more heavily used AWS container services is Amazon ECS (Elastic Container Service). While there is a trend toward using Fargate to simplify the setup and management of ECS clusters, many users still prefer using Amazon ECS with EC2 instances. It may not be as straightforward or efficient as AWS Fargate, but it offers more control over the underlying infrastructure.

In the most recent blog, we explored how Elastic Observability helps manage Amazon ECS with Fargate. However, this blog will review how to manage an Amazon ECS cluster with EC2 instances using Elastic Observability instead. 

In general, when setting up Amazon ECS-based clusters with EC2, you may or may not have access to the EC2 instances. This determines what you can use with Elastic Observability in monitoring your EC2-based ECS cluster. Hence, there are two components you can use in monitoring the EC2-based ECS cluster with Elastic Observability:

amazon ecs

As you can see in the diagram above, the two components are:

  1. Baseline setup: The elastic agent running the AWS integration is configured to obtain ECS metrics and logs from cloud watch. This agent runs on an instance that is not part of the ECS cluster because it allows you to see ALL ECS clusters and other AWS Services, such as EKS, RDS, and EC2. 

  2. Additional setup: If you have access to the EC2 instances in the ECS cluster, then you can run Elastic’s docker integration in each EC2 instance. This gives you significantly more details on the containers than AWS container insights. And it does not require AWS Cloudwatch, which can be fairly costly.

Using either just the baseline or the additional setup, you will have to set up AWS CloudWatch Container Insights for the ECS cluster.  However, the docker integration with the additional setup can provide additional information to the AWS CloudWatch Container Insights.

Hence, we will review how you can monitor the various components of an EC2-based ECS cluster:

  • EC2 instances in the ASG group
  • ECS services running in the ECS cluster
  • ECS tasks (containers)

Also, we will review how you can obtain metrics and logs from the ECS cluster with and without AWS Cloudwatch. We’ll show you how to use:

  • AWS CloudWatch Container Insights (from Cloudwatch) 
  • Docker metrics (non-Cloudwatch) 
  • Amazon ECS logs via Cloudwatch

Prerequisites and configuration

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

  • An account on Elastic Cloud and a deployed stack (see instructions here) — ensure that you have both.
  • A nginx container and a stress container — we will use these two basic containers to help highlight the load on the Elastic ECS Cluster.
  • An ECS EC2 Cluster in an Auto Scaling Group — ensure you have access in order to load up the Elastic agent on the EC2 instances, or you can create an AMI and use that as the baseline image for your ECS cluster.
  • An EC2 instance anywhere in your account that is not part of the ECS cluster and has public access (to send metrics and logs)

What will you see in Elastic Observability once it's all set up?

If you utilize the baseline configuration with ECS EC2 cluster configured with AWS CloudWatch Container Insights configured, the Elastic Agent configured with the following Elastic agent integrations:

  • ECS integration
  • EC2 integration
  • AWS Cloudwatch Integration with metrics and logging

Then you will be able to get the following information in Elastic dashboards:

  • Containers in the cluster (AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)
  • Services in the cluster (AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)
  • CPU and memory utilization of the ECS Cluster (Elastic Agent with ECS integration)
  • EC2 CPU and memory utilization of the instance in the cluster (Elastic Agent with EC2 integration)
  • CPU and memory utilization per container (via AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)
containers in cluster

If the additional configuration using Elastic agents with docker integration per ECS EC2 instance is used, you will be able to get a direct feed of metrics via docker. The following metrics can be viewed:

metrics graphs

Let’s see how to set this all up.

Setting it all up

Over the next few steps, I’ll walk through:

  • Getting an account on Elastic Cloud
  • Bringing up an ECS EC2 cluster and potentially setting up your own AMI
  • Setting up the containers nginx and a stress container
  • Setting up the Elastic agent with docker container integration on the ECS EC2 instances
  • Setting up the Elastic agent with AWS, Cloudwatch, and ECS integrations on an independent EC2 instance

Step 1: Create an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

free trial

Step 2: Set up an ECS Cluster with EC2 instances

When creating a cluster, you have two options when setting it up using the console:

  • Create a new ASG group where you will only be allowed to use the preloaded set of Amazon Linux (2 or 2023) based AMIs
  • Set up your own ASG Cluster prior to setting up the ECS Cluster and select this from the options. This option will give you more control over what Linux version and the ability to add things like Elastic agents in the AMI used for the instances in the ASG.

Regardless of either option, you will need to turn on Container Insights (see the bottom part of the image below).

infrastructure

Once the cluster is setup, you can go to AWS Cloudwatch where you should see Container Insights for your cluster:

container insights

Step 3: Set up Elastic agent with docker integration

Next, you will need to add an Elastic agent to each one of the instances. In the Elastic cloud, set up an Elastic policy with the docker and system integrations as such:

cluster policy

Next, add an agent for the policy, then copy the appropriate install script (in our case it was Linux since we were running Amazon Linux 2), and run it on every EC2 instance in the cluster:

add agent

Once this is added you should see agents in the fleet. Each agent will be on each EC2 instance:

fleet

If you decide to set up an ECS EC2 cluster with your own ASG and don’t use Amazon Lunix AMIs (2 or 2023 version), you will have to:

Step 4: Set up an Elastic agent with the AWS integration

From the integrations tab in Elastic Cloud, select AWS integration and select add agent. You will then have to walk through the configuration of the AWS integration.

At a minimum, ensure that you have the following configuration options turned on:

toggles

This will ensure that not only EC2 metrics and logs are ingested but that all CloudWatch metrics and logs are also ingested. ECS metrics and logs are stored in CloudWatch. 

If you want to ensure only logs from the specific ECS cluster are ingested, you can also restrict what to ingest by several parameters. In our setup, we are collecting only logs from Log Group with a prefix of /aws/ecs/containerinsights/EC2BasedCluster/.

cloudwatch

Once this policy is set up, add an agent like in Step 1.

add agent testing aws

However, this agent needs to be added to an EC2 instance which is independent of the ECS cluster.

Once installed, this agent will help pull in:

  • All EC2 instance metrics across your account (which can be adjusted in the integration policy)
  • Ingest AWS CloudWatch Container Insights from ECS
  • ECS metrics such as:
    • aws.ecs.metrics.CPUReservation.avg
    • aws.ecs.metrics.CPUUtilization.avg
    • aws.ecs.metrics.GPUReservation.avg
    • aws.ecs.metrics.MemoryReservation.avg
    • aws.ecs.metrics.MemoryUtilization.avg
    • More - see the full list here

Step 5: Setting up services and containers

In running this configuration, we used nginx and a stress container before we go into the task.

In order to initiate service and containers on ECS, you will need to set up a task for each of these containers. But more importantly, you will need to ensure that the roles for both of the following:

"taskRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRol"executionRoleArn":,

"arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",

have the following permissions:

permissions

Most importantly, you should ensure that this permission is added:

AmazonEC2ContainerServiceforEC2Role

It will ensure containers can be brought up on the EC2 instances in the cluster.

Once you have the right permissions, then set up the following tasks.

Here is the task JSON for NGINX:

{
    "family": "NGINX",
    "containerDefinitions": [
        {
            "name": "nginx",
            "image": "nginx: latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "nginx-80-tcp",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            }
        }
    ],
    "taskRoleArn": "arn:aws:iam::xxxxxx:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "256",
    "memory": "512",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}

Here is the task JSON for stress container:

{
    "family": "stressLoad",
    "containerDefinitions": [
        {
            "name": "stressLoad",
            "image": "containerstack/alpine-stress",
            "cpu": 0,
            "memory": 512,
            "memoryReservation": 512,
            "portMappings": [],
            "essential": true,
            "entryPoint": [
                "sh",
                "-c"
            ],
            "command": [
                "/usr/local/bin/stress --cpu 2 --io 2 --vm 1 --vm-bytes 128M --timeout 6000s"
            ],
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        }
    ],
    "taskRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "256",
    "memory": "512",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}

Once you have defined the tasks, ensure you bring up each service (one for each task) with the launch type of EC2:

environment

You should have two services running now.

ec2basedcluster

Step 6: Check on metrics and logs in Elastic Cloud

Go to Elastic Cloud and ensure that you are getting metrics and logs from the ECS Cluster. First, check to see if you are receiving metrics by viewing the built-in dashboard called [Metrics Docker] Overview.

Docker image

With some work on this dashboard by adding in container insight metrics and docker metrics, you should be able to see:

graphs

If you only have the ECS integration and the Elastic agent in Step 2, then you will need to create a new dashboard:

cluster

This dashboard can be set up with the following metrics:

  • Containers in the cluster (containerInsights via Elastic Agent and AWS Cloudwatch integration). Set up a TSVB panel using the following metric: aws.dimensions.ClusterName : "EC2BasedCluster" with aws.containerinsights.metrics.TaskCount.max
  • Services in the cluster (containerInsights via Elastic Agent and AWS Cloudwatch integration). Use the following configuration to setup the chart:
table
  • CPU and memory utilization of the ECS Cluster (Elastic Agent with ECS integration). Use the following configuration to set up both CPU and memory utilization charts:
line
  • EC2 CPU and storage utilization of the instance in the cluster (Elastic Agent with EC2 integration). Use the following configuration to set up both CPU and memory utilization charts:
bar vertical stacked
  • (Not shown): CPU and memory utilization per container (via containerInsights via Elastic Agent and AWS Cloudwatch integration)

Step 7: Look at logs from your ECS cluster

Since we set up AWS CloudWatch logs collection in Step 2, we can view these logs in Discover by filtering on the logs group arn /aws/ecs/containerinsights/EC2BasedCluster/.

logs

Summary

I hope you’ve gotten an appreciation for how Elastic Observability can help your AWS monitoring ECS service metrics. Here’s a quick recap of lessons and what you learned:

  • Elastic Observability supports ingesting and analysis of AWS ECS service metrics and the corresponding EC2 metrics through the AWS integration on the Elastic Agent. It’s easy to set up ingest from AWS Services via the Elastic Agent. 
  • Elastic Observability can also get container metrics via the Docker integration running on Elastic agents on each of the EC2 instances in the ECS EC2 auto scaling group.
  • Elastic has multiple out-of-the-box (OOTB) AWS service dashboards that can be used as baselines to get your own customized view.

Ready to get started? Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial