AWS ec2 metricsetedit

The ec2 metricset of aws module allows you to monitor your AWS EC2 instances, including cpu, network, disk and status. ec2 metricset fetches a set of values from Cloudwatch AWS EC2 Metrics.

We fetch the following data:

  • cpu.total.pct: The percentage of allocated EC2 compute units that are currently in use on the instance.
  • cpu.credit_usage: The number of CPU credits spent by the instance for CPU utilization.
  • cpu.credit_balance: The number of earned CPU credits that an instance has accrued since it was launched or started.
  • cpu.surplus_credit_balance: The number of surplus credits that have been spent by an unlimited instance when its CPUCreditBalance value is zero.
  • cpu.surplus_credits_charged: The number of spent surplus credits that are not paid down by earned CPU credits, and which thus incur an additional charge.
  • network.in.packets: The number of packets received on all network interfaces by the instance.
  • network.out.packets: The number of packets sent out on all network interfaces by the instance.
  • network.in.bytes: The number of bytes received on all network interfaces by the instance.
  • network.out.bytes: The number of bytes sent out on all network interfaces by the instance.
  • diskio.read.bytes: Bytes read from all instance store volumes available to the instance.
  • diskio.write.bytes: Bytes written to all instance store volumes available to the instance.
  • diskio.read.ops: Completed read operations from all instance store volumes available to the instance in a specified period of time.
  • diskio.write.ops: Completed write operations to all instance store volumes available to the instance in a specified period of time.
  • status.check_failed: Reports whether the instance has passed both the instance status check and the system status check in the last minute.
  • status.check_failed_system: Reports whether the instance has passed the system status check in the last minute.
  • status.check_failed_instance: Reports whether the instance has passed the instance status check in the last minute.
  • instance.core.count: The number of CPU cores for the instance.
  • instance.image.id: The ID of the image used to launch the instance.
  • instance.monitoring.state: Indicates whether detailed monitoring is enabled.
  • instance.private.dns_name: The private DNS name of the network interface.
  • instance.private.ip: The private IPv4 address associated with the network interface.
  • instance.public.dns_name: The public DNS name of the instance.
  • instance.public.ip: The address of the Elastic IP address (IPv4) bound to the network interface.
  • instance.state.code: The state of the instance, as a 16-bit unsigned integer.
  • instance.threads_per_core: The state of the instance (pending | running | shutting-down | terminated | stopping | stopped).

AWS Permissionsedit

Some specific AWS permissions are required for IAM user to collect AWS EC2 metrics.

ec2:DescribeInstances
ec2:DescribeRegions
cloudwatch:GetMetricData
cloudwatch:ListMetrics
sts:GetCallerIdentity
iam:ListAccountAliases

Dashboardedit

The aws ec2 metricset comes with a predefined dashboard. For example:

metricbeat aws ec2 overview

Configuration exampleedit

- module: aws
  period: 300s
  metricsets:
    - ec2
  access_key_id: '<access_key_id>'
  secret_access_key: '<secret_access_key>'
  session_token: '<session_token>'
  tags_filter:
    - key: "Organization"
      value: "Engineering"

tags_filter can be specified to only collect metrics with certain tag keys/values. For example, with the configuration example above, ec2 metricset will only collect metrics from EC2 instances that have tag key equals "Organization" and tag value equals to "Engineering".

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

Fieldsedit

For a description of each field in the metricset, see the exported fields section.

Here is an example document generated by this metricset:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "aws": {
        "ec2": {
            "cpu": {
                "credit_balance": 288,
                "credit_usage": 0.008857266666666667,
                "surplus_credit_balance": 0,
                "surplus_credits_charged": 0,
                "total": {
                    "pct": 0
                }
            },
            "diskio": {
                "read": {},
                "write": {}
            },
            "instance": {
                "core": {
                    "count": 1
                },
                "image": {
                    "id": "ami-067349b5a5143523d"
                },
                "monitoring": {
                    "state": "disabled"
                },
                "private": {
                    "dns_name": "ip-172-31-41-189.eu-north-1.compute.internal",
                    "ip": "172.31.41.189"
                },
                "public": {
                    "dns_name": "ec2-13-48-192-116.eu-north-1.compute.amazonaws.com",
                    "ip": "13.48.192.116"
                },
                "state": {
                    "code": 16,
                    "name": "running"
                },
                "threads_per_core": 2
            },
            "network": {
                "in": {
                    "bytes": 6557,
                    "bytes_per_sec": 21.856666666666666,
                    "packets": 51,
                    "packets_per_sec": 0.17
                },
                "out": {
                    "bytes": 8819,
                    "bytes_per_sec": 29.39666666666667,
                    "packets": 39,
                    "packets_per_sec": 0.13
                }
            },
            "status": {
                "check_failed": 0,
                "check_failed_instance": 0,
                "check_failed_system": 0
            }
        }
    },
    "cloud": {
        "account": {
            "id": "428152502467",
            "name": "elastic-beats"
        },
        "availability_zone": "eu-north-1b",
        "instance": {
            "id": "i-0853c3c24739b4696"
        },
        "machine": {
            "type": "t3.micro"
        },
        "provider": "aws",
        "region": "eu-north-1"
    },
    "event": {
        "dataset": "aws.ec2",
        "duration": 115000,
        "module": "aws"
    },
    "host": {
        "cpu": {
            "usage": 0
        },
        "id": "i-0853c3c24739b4696",
        "name": "i-0853c3c24739b4696",
        "network": {
            "egress": {
                "bytes": 8819,
                "packets": 39
            },
            "ingress": {
                "bytes": 6557,
                "packets": 51
            }
        }
    },
    "metricset": {
        "name": "ec2",
        "period": 10000
    },
    "service": {
        "type": "aws"
    }
}