We are happy to announce that Elastic Stack 7.5 brings support for the<a href="https://docs.microsoft.com/en-us/azure/azure-monitor/overview"> Azure Monitor</a> service in Metricbeat and <a href="https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-logs-overview">Azure activity logs</a> and <a href="https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/overview-reports#activity-reports">AD (Active Directory) activity reports</a> in Filebeat.
 <h2>Introducing the Azure module in Metricbeat</h2>The Azure Monitor service collects fundamental metrics and logs from a variety of sources in Azure, such as applications, subscriptions, tenants, and more. Using the new Azure Metricbeat module, users can retrieve all the metrics <a href="https://azure.microsoft.com/en-us/services/monitor/">Azure Monitor</a> exposes and forward them to any of the outputs that Metricbeat supports.
The Azure module includes three metricsets:
<ul>
	<li>The monitor metricset, which allows users to retrieve all types of metrics from specified resources. Using filters like metric namespace, name, aggregations and dimensions, users&nbsp; can limit the metrics to just that they are interested in.</li>
	<li>The compute_vm and compute_vm_scaleset metricsets allow users to collect all available metrics from virtual machines and virtual machine scale sets. The metric sampling duration set in the metricsets is 5 minutes so we recommend users to select a collecting interval (period) of 300s or multiples of 300s.</li>
</ul> <h3>Configuring the Metricbeat Azure module</h3>The following command enables the Azure module configuration:
<pre class="prettyprint">metricbeat modules enable azure
</pre>The module uses the Azure Monitor REST API to retrieve metrics, therefore, all requests must be authenticated with Azure Active Directory (Azure AD).&nbsp;
The required credentials for authorization are:
<ul>
	<li>client_id represents the unique identifier for the application (also known as Application Id)</li>
	<li>client_secret represents the client/application secret/key</li>
	<li>subscription_id represents the unique identifier for the azure subscription&nbsp;</li>
	<li>tenant_id represents the unique identifier of the Azure Active Directory instance</li>
</ul>One approach to retrieve these values is to create an Azure AD service principal. For a more detailed walk-through on generating an Azure service principal users can use the <a href="https://docs.microsoft.com/en-us/powershell/azure/create-azure-service-principal-azureps?view=azps-3.0.0&viewFallbackFrom=azps-2.7.0">Azure PowerShell tool</a> or they can create one via the <a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal">Azure portal</a>.
We recommend using a dedicated Azure service principal and making sure that the role assigned to the application contains reading permissions to the monitor service and the resources .&nbsp;
Example of a complete Azure module configuration (watch out for word wrap):
<pre class="prettyprint">- module: azure 
 metricsets: 
 - monitor 
 enabled: true 
 period: 300s 
 client_id: 'd3d1ab91-2707-4058-8274-05cd96339728' 
 client_secret: 'password' 
 tenant_id: 'd3d1ab91-2707-4058-8274-05cd96339728' 
 subscription_id: 'd3d1ab91-2707-4058-8274-05cd96339728' 
 refresh_list_interval: 600s 
 resources: 
 - resource_id: "subscriptions/d3d1ab91-2707-4058-8274-05cd96339728/resourceGroups/obs-infrastructure/providers/Microsoft.ApiManagement/service/apimanagement" 
 metrics: 
 - name: "Requests" 
 namespace: "Microsoft.ApiManagement/service" 
 aggregations: ["Maximum"] 
 timegrain: "PT1M" 
 dimensions: 
 - name: "Hostname" 
 value: "apimanagement.azure-api.net" 
 - name: ["Capacity", "Requests"] 
 namespace: "Microsoft.ApiManagement/service" 
 aggregations: ["Average"] 
 dimensions: 
 - name: "Location" 
 value: "West Europe" 
 - resource_group: "obs-infrastructure" 
 resource_type: "Microsoft.Compute/virtualMachines" 
 metrics: 
 - name: "*" 
 namespace: "Microsoft.Compute/virtualMachines" 
 - resource_query: "resourceType eq 'Microsoft.DocumentDb/databaseAccounts' and name eq 'dbaccount'" 
 metrics: 
 - name: ["DataUsage", "DocumentCount", "DocumentQuota"] 
 namespace: "Microsoft.DocumentDb/databaseAccounts" 
 aggregations: ["Average", "Total"]
</pre>We provide several options to select the monitored resources so users can choose between the following options:
<ul>
	<li>&nbsp;resource_id - the fully qualified ID of the resource which includes the resource name and resource type&nbsp;</li>
</ul>Since users can have a greater number of resources they would like to gather metrics on, in order to reduce the verbosity, they can also go for the following options:
<ul>
	<li>resource_group and resource_type - where metrics from all the resources of the selected resource type that are contained by the entered resource group will be returned.</li>
	<li>resource_query - users can enter a specific query to filter resources on (as defined by the $filter in the <a href="https://docs.microsoft.com/en-us/rest/api/resources/resources/list">Azure REST API resources documentation</a> ).</li>
</ul>We also offer options to filter the metrics to be collected:
<ul>
	<li>name - users can list specific metric names or enter "*" instead which will consider all metrics inside the namespace provided</li>
	<li>namespace - a list of the <a href="https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported">supported metrics and namespaces</a>, for&nbsp; specific guest os metrics or custom metrics will be categorized in additional namespaces</li>
	<li>aggregations - Azure Monitor stores all metrics at one-minute granularity intervals, but, to limit the number of raw values users will have to emit and pay for in Azure Monitor, the values are locally pre-aggregated. Each metric has a list of supported aggregations and one primary (most relevant) aggregation. Users can list the aggregations they would like to collect values of or ignore this option entirely which will return the primary aggregation value for the metric.</li>
	<li>dimensions - they are a list of name value pairs that help describe additional characteristics about the metrics. This configuration setting is optional as there are metrics that do not contain any dimensions.</li>
	<li>timegrain - represents the metric sampling duration, metrics can have several timegrain values assigned, if not value is entered a default one is provided.</li>
</ul>Azure resource details will be collected at each fetch call (period interval), this means a number of Azure REST calls will be executed each time. If the Azure users are adding/removing resources that match the configuration options while metricbeat is running, these resources will be reflected in Metricbeat as well. If users would like to reduce the number of API calls due performance and rate/ cost reasons they can use the refresh_list_interval configuration option and set an interval the list of resources is being updated.
For the compute_vm and compute_vm_scaleset metricsets, metrics will be collected from all the virtual machines and virtual machine scalesets in the entire subscription if no resource options are configured.
<pre class="prettyprint">- module: azure 
 metricsets: 
 - compute_vm 
 enabled: true 
 period: 300s 
 client_id: '${AZURE_CLIENT_ID:""}' 
 client_secret: '${AZURE_CLIENT_SECRET:""}' 
 tenant_id: '${AZURE_TENANT_ID:""}' 
 subscription_id: '${AZURE_SUBSCRIPTION_ID:""}' 
 refresh_list_interval: 600s
</pre><pre class="prettyprint">- module: azure
 metricsets:
 - compute_vm_scaleset
 enabled: true
 period: 300s
 client_id: '${AZURE_CLIENT_ID:""}'
 client_secret: '${AZURE_CLIENT_SECRET:""}'
 tenant_id: '${AZURE_TENANT_ID:""}'
 subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
 refresh_list_interval: 600s
</pre>To filter by resource group, users can add the following configuration:
<pre class="prettyprint">resources: 
 - resource_group: "test_resource_group"
</pre> <h3>Dashboards and visualizations</h3>These two metricsets come with dashboards and visualizations that help users view the most relevant metric information. Some dashboards expose extended metrics&nbsp; that users will need to explicitly enable in Azure.
To enable memory, ASP.NET, or SQLServer extended metrics, users can follow the <a href="https://docs.microsoft.com/bs-latn-ba/azure/cost-management/azure-vm-extended-metrics">documented steps for virtual machines</a> and the <a href="https://docs.microsoft.com/en-us/azure/azure-monitor/platform/collect-custom-metrics-guestos-resource-manager-vmss">documentation for virtual machine scalesets</a>.
The [Metricbeat Azure] Compute VMs overview dashboard offers a visual representation of the CPU percentage, incoming and outgoing traffic, average of bytes read/written from/to disk and the disk read/write IOPS. Additional memory metrics are displayed for the Windows vm (if the guest os metrics are collected).&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfbd91802209a6532/5defea216ac09b386d565219/azure-modules-blog-compute-vm-overview.jpg" data-sys-asset-uid="bltfbd91802209a6532" alt="[Metricbeat Azure] Compute VMs overview dashboard" style="display: block; margin: auto;">
The [Metricbeat Azure] VM Guest metrics overview dashboard provides visual information on the ASP.NET applications running, SQL Server connections, memory usage and read/write stats.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8210f17742cd3b85/5defea4b4e76c55d401b984b/azure-modules-blog-vm-guest-metrics.jpg" data-sys-asset-uid="blt8210f17742cd3b85" alt="[Metricbeat Azure] VM Guest metrics overview dashboard" style="display: block; margin: auto;">
The [Metricbeat Azure] VM Scale Sets overview dashboard offers a visual representation of the CPU percentage, incoming and outgoing traffic, average of bytes and operations read/written from/to disk.&nbsp; Additional memory metrics are displayed for the Windows VM (if the guest os metrics are collected).
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb78f9ad432c55efd/5defea611202185d397ac8b1/azure-modules-blog-vm-scale-sets.jpg" data-sys-asset-uid="bltb78f9ad432c55efd" alt="[Metricbeat Azure] VM Scale Sets overview dashboard" style="display: block; margin: auto;">
<h2>Introducing the Azure module in Filebeat</h2>With the Filebeat Azure module, users can retrieve Azure activity logs and AD activity reports.
A prerequisite for this functionality is that these logs should be exported to event hubs with an&nbsp; enabled kafka option as the module will collect the logs from these event hubs.
The activity logs provide insight into subscription-level events that have occurred in Azure. This includes a range of data, from Azure Resource Manager operational data to updates on Service Health events.
To export the activity logs to an event hub in the Azure Portal users can follow <a href="https://docs.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-export">these steps</a>.
The Azure module fileset that collects these logs is the activitylogs fileset.
The AD activity reports include the sign-in logs which provide information about the usage of managed applications and user sign-in activities and the audit logs which provide traceability through logs for all changes done by various features within Azure AD.
To export the Azure AD reports to an event hub in the Azure Portal users can follow <a href="https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/tutorial-azure-monitor-stream-logs-to-event-hub">these steps</a>.&nbsp;
The Azure module filesets that collect these two types of logs are named signinlogs and auditlogs, respectively.
<h3>Configuring the Azure module in Filebeat</h3>The following command enables the Azure module configuration :
<pre class="prettyprint">filebeat modules enable azure
</pre>Example configuration:
<pre class="prettyprint">- module: azure 
 activitylogs: 
 enabled: true 
 var: 
 namespace: "eventhubs.servicebus.windows.net:9093" 
 eventhub: ["insights-operational-logs"] 
 consumer_group: "$Default" 
 connection_string: "" 
 auditlogs: 
 enabled: true 
 var: 
 namespace: "eventhubs.servicebus.windows.net:9093" 
 eventhub: ["insights-logs-auditlogs"] 
 consumer_group: "$Default" 
 connection_string: "" 
 signinlogs: 
 enabled: true 
 var: 
 namespace: "eventhubs.servicebus.windows.net:9093" 
 eventhub: ["insights-logs-signinlogs"] 
 consumer_group: "$Default" 
 connection_string: ""
</pre><ul>
	<li>namespace contains the Azure event hub namespace FQDN followed by the kafka protocol</li>
	<li>eventhub contains the event hub name the logs are exported to</li>
	<li>consumer_group contains the consumer group name, we advise having a dedicated consumer group for the filebeat module.</li>
	<li>connection_string contains the connection string required to communicate with Event Hubs, users can find it inside the <a href="https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string">SAS policy area</a>.&nbsp;</li>
</ul> <h3>Dashboards and visualizations</h3>The Azure module comes with several visualizations and dashboards where users can view resource activity and monitor signin and audit events.
The [Filebeat Azure] Cloud Overview offers a visual representation of the total number of users/resources/resource groups/subscriptions mentioned in the logs. The dashboards comes with several overviews of the activity, sign in alert and service health levels.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbb6e357defe33551/5df043735f0d0b37d595108e/azure-modules-blog-cloud-overview.jpg" data-sys-asset-uid="bltbb6e357defe33551" alt="[Filebeat Azure] Cloud Overview dashboard" style="background-color: initial; display: block; margin: auto;">
The [Filebeat Azure] Alerts Overview provides an outline of the activity alerts count and overview and lists the most affected resources.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt287a90d4dd2414ab/5defeaab61829a38dad2cf9c/azure-modules-blog-alerts-overview.jpg" data-sys-asset-uid="blt287a90d4dd2414ab" alt="[Filebeat Azure] Alerts Overview dashboard" style="display: block; margin: auto;">
The [Filebeat Azure] User Activity provides an outline of the user list and activity and top resources and resource types affected.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdd53decda181d64c/5df042d78c8da537d52c3d1f/azure-modules-blog-user-activity.jpg" data-sys-asset-uid="bltdd53decda181d64c" alt="[Filebeat Azure] User Activity dashboard" style="background-color: initial; display: block; margin: auto;">
<h2>What’s next</h2>With the monitor metricset users can already have access to all metrics exposed by the Azure Monitor.&nbsp;
To further simplify access to this data we will continue adding more service specific integrations with Azure and enriching these modules with new metricsets and filesets that will monitor the most relevant Azure services. We encourage you to test these new modules out, either in your own cluster or in a <a href="https://www.elastic.co/elasticsearch/service">free trial of Elasticsearch Service</a>.
We are very interested in your feedback so let us know what you think about this first step at supporting Azure with Beats in the <a href="https://discuss.elastic.co/c/beats">Beats forum</a>.

azure-modules-blog-banner-dark.jpg

azure-modules-blog-thumb.jpg

Monitoring Azure activity logs, reports, and metrics with new Beats modules

New Beats modules for metrics from Azure Monitor, Azure activity logs, and AD activity reports

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Mariana Dima

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Mariana Dima

New Beats modules for metrics from Azure Monitor, Azure activity logs, and AD activity reports

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS