<p>Starting with Metricbeat 7.2, we introduced the new <code>cloudwatch</code> metricset in the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.4/metricbeat-module-aws.html">AWS module</a> to monitor various AWS services. Amazon CloudWatch provides data and insights to better monitor performance, utilization and health of their applications and services. Instead of adding a new metricset for each service in AWS module, the CloudWatch metricset can be used to collect monitoring metrics from any services that publishes CloudWatch metrics.
</p><p>On top of the basic functionalities of the <code>cloudwatch</code> metricset in 7.2, we added the ability to specify statistic methods and enable tag collections to give users a better out of the box experience and flexibility in upcoming 7.4 release. This also enables us to extend metricsets in the aws module using the Metricbeat light module. Light modules are a new way of defining Metricbeat modules or metricsets based on existing implementations. Instead of writing Golang code, it uses configuration files based on existing metricset in Metricbeat. Please see introducing Metricbeat light module for more details and great examples.
</p><h2>Introduction to CloudWatch metricset</h2><p>Running the <code>cloudWatch</code> metricset requires settings in AWS account, AWS credentials, and a running Elastic Stack. Elastic Stack includes Elasticsearch for storing and indexing the data, and Kibana for data exploration. For this you can either create a cluster in <a href="https://www.elastic.co/elasticsearch/service">Elasticsearch Service</a> on <a href="https://www.elastic.co/cloud/">Elastic Cloud</a> or set up the <a href="https://www.elastic.co/guide/en/elastic-stack-get-started/7.4/get-started-elastic-stack.html">Elastic Stack</a> on your local machine.
</p><h3>1. Create an IAM policy</h3><p>An <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html">IAM policy</a> is an entity that defines permissions to an object within your AWS environment. Creating a customized IAM policy for Metricbeat with specific permissions is needed. Please see <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html">Creating IAM Policies</a> for more details. After Metricbeat IAM policy is created, you need to add this policy to the IAM user which provided the credentials in the previous step.
</p><p>With the <code>cloudWatch</code> metricset, four permissions need to be added into Metricbeat IAM policy:
</p><ul>
	<li><code>ec2:DescribeRegions</code></li>
	<li><code>ec2:DescribeInstances</code></li>
	<li><code>cloudwatch:ListMetrics</code></li>
	<li><code>cloudwatch:GetMetricData</code></li>
	<li><code>tags:getResources</code></li>
</ul><h3>2. Enable the AWS module</h3><p>In the out-of-box configuration of Metricbeat, only the <code>system</code> module is enabled by default, so you will need to explicitly enable the AWS module. The following command enables the AWS configuration in the <code>modules.d</code> directory on MacOS and Linux systems:
</p><pre class="prettyprint">metricbeat modules enable aws
</pre><h3>3. Set AWS credentials in the config file</h3><p>To configure AWS credentials, users can put the credentials into the Metricbeat module configuration or using environment variables to pass them. In Metricbeat 7.4, the ability to load AWS credentials from a shared credentials file is added into aws module. Shared credentials file is at <code>C:\Users\&lt;yourusername&gt;\.aws\credentials&lt;/yourusername&gt;</code> for Windows and <code>~/.aws/credentials</code> for Linux, MacOS, or Unix. Users can specify the profile name using parameter <code>credential_profile_name</code> in aws module config.  For more details on AWS credentials types and supported formats, please see <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.4/metricbeat-module-aws.html#aws-credentials-config">AWS credentials configuration</a> for more detail.
</p><h3>4. Configure CloudWatch metricset</h3><p>There are several <code>cloudWatch</code> metricset specific parameters in configuration: <code>namespace</code>, <code>name</code>, <code>dimensions</code>, <code>statistic</code> and <code>tags.resource_type_filter</code>.
</p><ul>
	<li><code>namespace</code>: A namespace in AWS CloudWatch is a container for metrics from a specific application or service. Each service has its own namespace, for example Amazon EC2 uses AWS/EC2 namespace and Amazon Elastic Block Storage uses AWS/EBS namespace. Please see the full list of services and namespaces that publish CloudWatch metrics for more details.</li>
	<li><code>name</code>: Users can specify what are the specific CloudWatch metrics need to be collected per namespace.</li>
	<li><code>dimensions</code>: Dimensions are used to refine metrics returned for each instance. For example, InstanceId, ImageId and InstanceType all can be used as dimensions to filter data requested from Amazon EC2.</li>
	<li><code>statistic</code>: Users can specify one or more statistic methods for each CloudWatch metric setting. By default, average, count, maximum, minimum and sum will all be collected for each metric.</li>
	<li><code>tags.resource_type_filte</code>: Tags for resources will not be collected unless this parameter is set. Each resource in AWS has a specific resource type and the common format is <em>service[:resourceType]</em>. Please see <a href="https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/API_GetResources.html#resourcegrouptagging-GetResources-request-ResourceTypeFilters">resource type filters</a> for more details.</li>
</ul><p>With these parameters, users can use <code>cloudWatch</code> metricset to monitor one or more specific namespaces, one or more specific CloudWatch metrics, different metrics with different statistic methods, etc. For example, with the configuration below, three sets of metrics will be collected from CloudWatch: AWS SNS metrics with Sum statistic method, average `HealthCheckPercentageHealthy`, and average `SSLHandshakeTime` metrics from AWS Route53, AWS EC2 metrics for instance i-1234 with all default statistic methods.
</p><pre class="prettyprint">- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  credential_profile_name: test-metricbeat
  metrics: 
    - namespace: AWS/SNS
      statistic: ["Sum"]
    - namespace: AWS/Route53
      name: ["HealthCheckPercentageHealthy", "SSLHandshakeTime"]
      statistic: ["Average"]
    - namespace: AWS/EC2
      tags.resource_type_filter: ec2:instance
      dimensions:
  - name: InstanceId
    value: i-1234
</pre><h3>5. Start Metricbeat</h3><p>Usage running on mac and linux system as an example:
</p><pre class="prettyprint">metricbeat -e
</pre><h2>Creating new metricsets using light modules</h2><p>In 7.4, the <code>elb</code> metricset for AWS Elastic Load Balancing has been introduced using <a href="https://www.elastic.co/blog/introducing-metricbeat-light-modules">light modules</a>, based on <code>cloudwatch</code> metricset. With the metricset-specific parameters for <code>cloudwatch</code>, we were able to define the <code>elb</code> metricset to collect different metrics with different statistics. In <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-cloudwatch-metrics.html">classic load balancer metrics documentation</a>, the most useful statistic method is specified for each metric. For example, for metric Latency, the most useful statistic is Average; for metric BackendConnectionErrors, the most useful statistic is Sum and Average/Minimum/Maximum are reported per load balancer node which are not typically useful. Therefore, in <a href="https://github.com/elastic/beats/blob/7.4/x-pack/metricbeat/module/aws/elb/manifest.yml">manifest.yml</a>, which defines the new elb metricset using light module, we have different metrics specified with different statistics.
</p><p>To run the <code>elb</code> metricset, it’s the same as running other metricsets in Metricbeat:
</p><pre class="prettyprint">- module: aws
  period: 300s
  metricsets:
    - elb
  credential_profile_name: test-metricbeat
</pre><p>This metricset also includes a predefined dashboard that displays some of the metrics, such as:
</p><ul>
	<li><code>aws.metrics.HealthyHostCount.avg</code> for the number of healthy instances registered with load balancer.</li>
	<li><code>aws.metrics.UnheathyHostCount.avg</code> for the number of unhealthy instances registered with load balancer.</li>
	<li><code>aws.metrics.RequestCount.sum</code> for the number of requests completed or connections made during the specific interval.</li>
	<li><code>aws.metrics.Latency.avg</code> for the total time elapsed, in seconds. For HTTP listener, it’s from the time the load balancer sent the request to a registered instance until the instance started to send the response headers. For TCP listener, it's time for the load balancer to successfully establish a connection to a registered instance.</li>
	<li><code>aws.metrics.BackendConnectionErrors.sum</code> for the number of connections that were not successfully established between the load balancer and the registered instances.</li>
	<li><code>aws.metrics.HTTPCode_Backend_2XX.sum</code> for the number of HTTP response codes 2XX generated by registered instances.</li>
	<li><code>aws.metrics.HTTPCode_Backend_4XX.sum</code> for the number of HTTP response codes 4XX generated by registered instances.</li>
	<li><code>aws.metrics.HTTPCode_ELB_5XX.sum</code> for the number of HTTP 5XX server error codes generated by the load balancer.</li>
</ul><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9d82a7e3c6a898bb/5d82486839b84c70851d5dcd/Screen_Shot_2019-09-18_at_9.08.05_AM.png" alt="Kibana dashboard for AWS Classic ELB" "="" style="display: block; margin: auto;">
</p><p>With this dashboard, users can check overall healthy host count and unhealthy host count, latency and errors. Users can also use the AWS Region Filter to only display load balancer data from one specific region to get a closer look.
</p><h2>What’s next</h2><p>With the <code>cloudWatch</code> metricset, users can monitor different AWS services without waiting for individual metricset to be added into Metricbeat. For Beats contributors, <a href="https://www.elastic.co/blog/introducing-metricbeat-light-modules">Writing Metricbeat modules based on other modules: Introducing light modules</a> introduced a faster and easier way to create new metricsets for more useful AWS services. We also highly encourage users to create light modules by converting a <code>cloudWatch</code> metricset configuration into a metricset for a specific AWS service that others can benefit as well. If you have any questions or feedback, please feel free to post on <a href="https://discuss.elastic.co/c/beats">Beats discuss forum</a>!
</p><p>You can start capturing and analyzing your AWS CloudWatch metrics with the <code>cloudWatch</code> metricset, today, and can ship metrics to Elasticsearch running on <a href="https://www.elastic.co/cloud/">Elastic Cloud</a>, or to a cluster running the <a href="https://www.elastic.co/guide/en/elastic-stack-get-started/7.4/get-started-elastic-stack.html">default distribution</a> of the Elastic Stack.
</p>

cloud-watch-blog-banner.jpg

cloud-watch-blog-thumb.jpg

Monitoring AWS services using the CloudWatch metricset

<p>Starting with 7.0, we introduced a new module in Metricbeat to monitor Amazon Web Services. At the time of the initial release in 7.0.0-beta1, this <a href="/guide/en/beats/metricbeat/7.0/metricbeat-module-aws.html">AWS module</a> provides ability to collect AWS EC2 monitoring metrics from Cloudwatch using Metricbeat and stores them in a metric set called <code>ec2</code>. These metrics provide visibility into all EC2 instances across regions without the need to install Metricbeat directly onto the instances.
</p>
<h2>Background</h2>
<p>Amazon Elastic Compute Cloud (EC2) instances are virtual servers that are created on the AWS platform to provide users with secure and resizable compute capacity in the public cloud. EC2 is arguably the most critical service out of hundreds provided by AWS -- virtually everyone needs access to raw compute resources.
</p>
<p>Monitoring EC2 instances centrally to help users detect and debug issues is challenging. Metricbeat can already collect detailed resource metrics using the <code>system</code> module, but this requires installing an instance of Metricbeat on each of your EC2 instances. While the metrics it collects are very detailed and granular, sometimes you just want a quick way to get visibility into resource usage of all your EC2 instances at a high level.
</p>
<p>This is where AWS Cloudwatch comes in. Cloudwatch automatically collects and centralizes basic resource utilization metrics from all your EC2 instances. This data is collected at a 5-minute granularity by default, and you can sign up for <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html">detailed monitoring</a> to get higher granularity and more advanced metrics.
</p>
<h2>Configuring Metricbeat AWS </h2>
<p>Before getting started with Metricbeat, you will need a running Elastic Stack, which includes Elasticsearch for storing and indexing the data and Kibana for data exploration. For this you can either create a cluster in <a href="/cloud/elasticsearch-service">Elasticsearch Service</a> on <a href="/cloud/">Elastic Cloud</a> or <a href="/guide/en/elastic-stack-get-started/7.0/get-started-elastic-stack.html">set up the Elastic Stack</a> on your local machine.
</p>
<div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/ooYanWgT2svNAUHH9xjKXW.jpg" data-uuid="ooYanWgT2svNAUHH9xjKXW" data-v="4" data-type="inline">
</div>
<p>Once you loaded Kibana, you can navigate to the <strong>Add Data UI</strong> from the home screen and follow the AWS module setup instructions from there.<br>
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt077d32a6001cc141/5c8ff8026f69e6812ec2e4d0/download" data-sys-asset-uid="blt077d32a6001cc141" alt="Deploying Metricbeat for AWS EC2" "="">
</p>
<p>This blog will walk you through them in more detail below.
</p>
<h2>Enable the AWS module</h2>
<p>In the out-of-box configuration of Metricbeat, only the <code>system</code> module is enabled by default, so you will need to explicitly enable the AWS module. The following command enables the AWS configuration in the <em>modules.d</em> directory on MacOS and Linux systems:
</p>
<pre class="prettyprint">./metricbeat modules enable aws
</pre>
<h2>Set AWS credentials in the config file</h2>
<p>There are two different kinds of AWS credentials which can be used: <a href="https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys">access keys</a> and <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html">temporary security credentials</a>. Access key ID and secret access key are the two parts of access keys. They are long-term credentials for an IAM user or the AWS account root user to authenticate when making AWS API calls. Temporary security credentials consist of an access key ID, a secret access key, and a security token, which have a limited lifetime.
</p>
<p>Because temporary security credentials are short term, after they expire, you need to generate new ones and modify <code>modules.d/aws.yml</code> config file with the new credentials. Unless <a href="/guide/en/beats/metricbeat/6.6/_live_reloading.html">live reloading</a> feature is enabled for Metricbeat, you need to manually restart Metricbeat after updating the config. <em>This will cause data collection to stop, if the config file is not updated with new credentials before the old ones expire.</em> As a result, for Metricbeat, we recommend you to use access keys in config file to enable the AWS module making AWS API calls without have to generate new temporary credentials and update the config frequently.
</p>
<h2>Create an IAM policy</h2>
<p>An <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html">IAM policy</a> is an entity that defines permissions to an object within your AWS environment. Creating a customized IAM policy for Metricbeat with specific permissions is needed. Please see <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html">Creating IAM Policies</a> for more details. After Metricbeat IAM policy is created, you need to add this policy to the IAM user which provided the credentials in the previous step.
</p>
<p>With the EC2 metric set, four permissions need to be added into Metricbeat IAM policy:
</p>
<ul>
	<li><code>ec2:DescribeRegions</code></li>
	<li><code>ec2:DescribeInstances</code></li>
	<li><code>cloudwatch:ListMetrics</code></li>
	<li><code>cloudwatch:GetMetricData</code></li>
</ul>
<h2>Start Metricbeat</h2>
<p>Usage&nbsp;running on mac and linux system as an example:
</p>
<pre class="prettyprint">./metricbeat -E setup.dashboards.enabled=true -e
</pre>
<p><em>Note:</em> If you are running Metricbeat from a virtual machine on a cloud instance, such as GCP, AWS, or Azure, you will want to remove or comment out the <code>add_cloud_metadata</code> entry in metricbeat.yml.
</p>
<p>Once the EC2 metric set is enabled in Metricbeat with proper AWS credentials, Metricbeat will periodically fetch information about each instance and collect EC2 monitoring metrics from AWS Cloudwatch. The default collection period for the EC2 metric set is 300 seconds. If <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html">detailed monitoring</a> for instances are enabled, the EC2 metric set collection period should be changed to 60s, since that paid tier provides higher-granularity metrics collection. The collection period is set in <code>modules.d/aws.yml</code> file, for example:
</p>
<pre class="prettyprint">- module: aws
  period: 60s
  metricsets:
    - "ec2"
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
  session_token: '${AWS_SESSION_TOKEN:""}'
  default_region: '${AWS_REGION:us-west-1}'
</pre>
<p>The EC2 metric set will discover new instances automatically and start collecting metrics from them. If an EC2 instance is stopped or terminated, the EC2 metric set will discover the instance state change in next collection cycle and send the new state metric to Elasticsearch.
</p>
<h2>Using the Kibana AWS module dashboard</h2>
<p>The EC2 metric set includes a predefined dashboard, called <strong>[Metricbeat AWS] EC2 Overview</strong>. By running the setup command when you started Metricbeat, you have automatically set up these dashboards in Kibana. For more information on this, please see <a href="/guide/en/beats/metricbeat/6.6/load-kibana-dashboards.html">Set up the Kibana dashboards</a>.
</p>
<p>This dashboard is an overview of AWS EC2 instances. It shows state and status checks for instances, as well as CPU utilization, network, and disk I/O data. There are also filters that you can apply to narrow the data to a specific region, an availability zone or a machine type.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt090c04cc347ff8d1/5c8ff81849324e0c3a98c52e/download" data-sys-asset-uid="blt090c04cc347ff8d1" alt="Kibana dashboard for AWS EC2" "="">
</p>
<p>With this dashboard, you can detect issues and make sure every instance is optimized for usage. For example, if you are seeing continuously high CPU utilization and performance issues from the same instance, this possibly indicates that CPU is the bottleneck and this instance needs more CPU power. Also, if you see an instance with low CPU utilization for a long time, this means the instance is oversized on CPU, and possibly can be scaled down or consolidated with another instance.
</p>
<h2>What’s next</h2>
<p>Having a centralized location to visualize monitoring metrics from different AWS services and regions allows you to optimize resources usage, detect and fix issues faster. Give it a <a href="/guide/en/beats/metricbeat/7.0/metricbeat-module-aws.html">try</a>, let us know what you think, give us feedback, and tell us how you are using this in our <a href="https://discuss.elastic.co/c/beats">Beats forum</a>.
</p>The EC2 metricset in the AWS module is just the beginning! We are already hard at work on adding more metric sets to monitor other commonly used services, such as Amazon S3, SQS, RDS, and so on. If you have an opinion on what we should tackle next or want to contribute, please do not hesitate to engage in the <a href="https://github.com/elastic/beats/issues">Beats Github repository</a>!

blog-banner-generic-teal.png

aws-metrics-thumb.png

Monitoring AWS EC2 using Metricbeat

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-kaiyan-sheng-300x300.jpg

Software Engineer

Kaiyan Sheng

<p>Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.</p>

Newsletter GDPR Text - Ukrainian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

Intentionally Blank

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。</p>

Newsletter GDPR Text - TW

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p><p>By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.</p>

GDPR Text + Google Cloud

<p>Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.</p>

Newsletter GDPR Text - Italian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Kaiyan Sheng

Software Engineer, Elastic

Monitoring AWS services using the CloudWatch metricset

Monitoring AWS EC2 using Metricbeat