Native Azure integrationedit

The Elastic Cloud native Azure integration allows you to deploy managed instances of the Elastic Stack directly in Azure, through the Azure integrated marketplace. The integration brings the following benefits:

  • Easy deployment for managed Elastic Stack instances

    Elastic Stack instances managed by Elastic can be deployed directly from the Azure console. This provides the complete Elastic Stack experience with all commercial features.

  • Integrated billing

    You are billed directly to your Azure account; no need to configure billing details in Elastic.

  • Easy consolidation of your Azure logs in Elastic

    Use a single-step setup to ingest logs from your Azure services into the Elastic Stack.

Check the following sections to learn more:

Getting startededit

How do I get started with Elastic Cloud?

Elastic Cloud is available as an offering through the Azure console.

Prerequisites

There are a few requirements to check before setting up an Elastic Cloud deployment:

  • You first must be configured in Azure as a subscription Owner. Check Assign a subscription administrator in the Azure documentation for steps to assign an Owner role.
  • You cannot use an email address that already has an Elastic Cloud account. If your Azure account address is currently used for an Elastic Cloud account, either use a different Azure account to set up the Elasticsearch resource, or contact the Elastic Support Team for assistance.
  • You must have a credit card present on your Azure subscription. If you have a non-payment subscription, such as a Virtual Studio Subscription, you won’t be able to create an Elastic Cloud deployment. Refer to the Azure Purchase errors troubleshooting documentation for more information.
  • In order to single sign-on into your Elastic Cloud deployment from Azure you need to request approval from your Azure administrator.

    A UI dialog indicating that the Elasticsearch app in Azure requires Admin approval for SSO access.

Getting started

Once you’re configured in Azure as a subscription owner, you can access Elastic Cloud in two ways:

  1. Search for elastic in the Azure Search bar. On the results page, the Elasticsearch link under the Services menu will open a page that guides you through deploying the Elastic Stack.

    Search results showing Elasticsearch at the top of a list under Services
  2. Navigate to the Azure Marketplace page. There, you’ll find the Elastic native integration listed as Elasticsearch (Elastic Cloud). Open the link and follow the on-screen steps to deploy the Elastic Stack.

When you create an Elastic Cloud deployment, an Elastic Stack cluster is created for you. The size of this deployment is 16GB of RAM and 480GB of storage, across two availability zones for redundancy. The size of the deployment, both RAM and storage, can be changed directly in the Elastic console. Usage charges are based on the size of your deployment so ensure that your instance is sized efficiently. The deployment defaults to the latest available version of the Elastic Stack. Check our Version policy to learn more about when new versions are made available and old versions are removed from service.

What is the pricing for this offer?

Pricing is pay-as-you-go per hour for each Elastic Cloud deployment that you have created. Note that there is no free trial period for the offering. Charges are applied to your Azure bill at the end of the month. You can use the Elastic Cloud Pricing Calculator to size a deployment and view the corresponding hourly rate.

Elastic charges include:

Which Azure regions are supported?
Here is the list of available Azure regions supported in Elastic Cloud.
Which Elastic Cloud subscription levels are available?
Elastic Cloud offers a number of different subscription levels. Your Elastic Cloud subscription will default to the Enterprise subscription, giving you immediate access to advanced Elastic Stack features like machine learning, as well as premium support response time SLAs.
How can I change my Elastic Cloud subscription level?

You can modify your subscription level on the billing page in the Elastic console.

  1. Open the Elastic Cloud list deployments page in Azure. From here, you can view any existing Elastic Cloud deployments that you have created.
  2. Select a deployment to open the deployment overview page.
  3. Select the Advanced Settings link to access your deployment in the Elastic Cloud console.

    The deployment overview page in Azure with the change my subscription link highlighted
  4. In the Elastic Cloud console, select your account avatar icon at the top of the page, and then choose Account & Billing.
  5. Select the Billing tab and choose Change my subscription.

    The Elastic Account Billing page with Advanced Settings highlighted
  6. Select the subscription level that you’d like.

    The Update Subscription page showing Standard
Can I subscribe using an email address from another Elastic account?
Your email address can be associated with only one Elastic account. For a workaround, check Sign up using an email address from another Cloud account.
Is the Elasticsearch native integration connected with Azure user management?

No. Elastic is not currently integrated with Azure user management. Azure users who deploy Elasticsearch on Azure can view and manage their own cluster through the Cloud console, but other Azure users in the same tenant cannot access clusters through the Cloud console other than those that they themselves created.

In the Cloud console, when you try to access resources such as Elasticsearch, Kibana, Enterprise Search, or APM in a deployment that was created by another Azure user, the following error is shown:

Error message displayed in the Elastic Cloud console: To access the resource {resource-name}

However, you can share deployment resources directly with other Azure users by configuring Active Directory single sign-on with the Elasticsearch cluster.

I already have an Elastic Cloud account, can I use this integration?
Yes. If you already have an Elastic Cloud account with the same email address as your Azure account you may need to contact support@elastic.co.
Can I sign up for an Elastic Cloud trial account and then convert to the Azure Elasticsearch native integration?

Yes. You can start a free Elasticsearch Service trial and then convert your account over to Azure. There are a few requirements:

  • Make sure that when you create deployments in the trial account you specify Azure as the cloud provider.
  • To convert your trial to the Azure marketplace you need to create a deployment in the Azure console. You can just delete the new deployment if you don’t need it. After you create the new deployment your marketplace subscription will be ready.
  • Any deployments that you create during your trial won’t show up in the Azure console, since they weren’t created in Azure, but they are still accessible through the Elasticsearch Service Console and you will be billed for their usage.
Does Elasticsearch get deployed into my tenant in Azure?
No. Elasticsearch resources get deployed in an Azure tenant that is managed by Elastic. This tenant, and the management capabilities associated with it, are the same that are used to run Elastic’s managed service, which also allows users to deploy on Azure.
What Azure tenant information does Elastic have access to?

After you subscribe to Elastic Cloud through the native Azure integration, Elastic has access to the following Azure tenant information:

  • Data defined in the marketplace Saas fulfillment Subscription APIs.
  • The following additional data:

    • Marketplace subscription ID
    • Marketplace plan ID
    • Azure Account ID
    • Azure Tenant ID
    • Company
    • First name
    • Last name
    • Country

Elastic can also access data from native integration features, including resource and activity log data. This data is available to Elastic only if you enable it. By default, Elastic does not have access to this information.

What other methods are available to deploy Elasticsearch?

Currently, Elasticsearch can be deployed on Azure using any of the following methods:

Official Azure SDKs

Other methods

How do I migrate my data from the classic Azure marketplace account to the native integration?

It’s easy. You’ll first need to create a new account configured with Azure native integration, and then you can perform the migration as follows:

  1. From your classic Azure marketplace account, navigate to your deployment and configure a custom snapshot repository using Azure Blog Storage.
  2. Using the newly configured snapshot repository, create a snapshot of the data that you want to migrate.
  3. Navigate to Azure and log in as the user that you want to manage the Elasticsearch resources.
  4. Before proceeding, ensure your new account is configured according to the prerequisites.
  5. Create a new Elasticsearch resource for each existing deployment that needs to be migrated from the classic Azure account.
  6. In the new Elasticsearch resource, follow the steps in Restore from a snapshot to register the custom snapshot repository from Step 1.
  7. In the same set of steps, restore the snapshot data from the snapshot repository that you registered.
  8. Confirm that your data has moved successfully into your new Elasticsearch resource on Azure.
  9. To remove the old Azure subscription and the old deployments, go to the Azure SaaS page and unsubscribe from the Elastic Cloud (managed Elasticsearch) marketplace subscription. This action will trigger the existing deployments to be terminated.

Managing your Elastic Cloud deploymentedit

What is included in my Elastic Cloud deployment?

Each Elastic Cloud deployment includes:

  • An Elasticsearch cluster
  • A Kibana instance which provides data visualization and a front-end for the Elastic Stack
  • An APM server that allows you to easily collect application traces
  • An Enterprise Search instance that allows you to easily build a search experience with an intuitive interface
How can I access my Elastic Cloud deployment?

There are a few ways. First, you need to navigate to the deployment overview page in Azure:

  1. Open the Elastic Cloud list deployments page in Azure. From here, you can view any existing Elastic Cloud deployments that you have created.
  2. Select a deployment to open the deployment overview page.

    The deployment overview page in Azure

    You now have a few options to access your deployment:

    • Elasticsearch endpoint - the URL for the Elasticsearch cluster itself
    • Kibana endpoint - the UI for the Elastic Stack, a great way for new users to get started
    • Elastic Cloud - Open the Advanced Settings link to access your deployment in the Elastic Cloud console, where you can make changes such as resizing or upgrading your deployment.
How can I modify my Elastic Cloud deployment?

You can modify your Elastic Cloud deployment in the Elastic Cloud console, which you can access from the Azure UI through the Advanced Settings link on the deployment overview page. In the Elastic Cloud console you can perform a number of actions against your deployment, including:

How can I delete my Elastic Cloud deployment?

You can delete your deployment directly from the Azure console. The delete operation, when evoked from Azure, will perform clean-up activities in the Elastic console to ensure any running components are removed, so that no additional charges occur.

The deployment overview page in Azure with the Delete button highlighted

Configuring logs and metricsedit

How do I monitor my existing Azure services?

The Elastic Cloud native Azure integration greatly simplifies logging for Azure services with the Elastic Stack. This integration supports the easy setup of Azure platform logs, including:

The following log types are not supported as part of this integration:

  • Azure tenant logs
  • Logs from Azure compute services, such as Virtual Machines

If your Azure resources and Elastic deployment are in different subscriptions, before creating diagnostic settings confirm that the Microsoft.Elastic resource provider is registered in the subscription in which the Azure resources exist. If not, register the resource provider following these steps:

  1. In Azure, navigate to Subscriptions → Resource providers.
  2. Search for Microsoft.Elastic and check that it is registered.

If you already created diagnostic settings before the Microsoft.Elastic resource provider was registered, delete and add the diagnostic setting again.

In the Azure console, you can configure ingestion of Azure logs into either a new or existing Elastic Cloud deployment:

  • When you create a new deployment, from the Logs & metrics tab in Azure you can specify the log type and a key/value tag pair. Any Azure resources that match on the tag value will automatically send log data to the Elastic Cloud deployment, once it’s been created.
The Logs & Metrics tab on the Create Elastic Resource page
  • For existing deployments, you can configure Azure logs from the deployment overview page in the Azure console.

Note that following restrictions for logging:

  • Only logs from non-compute Azure services are ingested as part of the configuration detailed in this document. Logs from compute services, such as Virtual Machines, into the Elastic Stack will be added in a future release.
  • The Azure services must be running in one of the following regions:

    • Eastus
    • Eastus2
    • Westus2
    • Centralus
    • Southcentralus
    • Uksouth
    • Francecentral
    • Japaneast
    • Southeastasia
    • Westeurope
    • Australiaeast
    • Northeurope

All regions will be supported in the future.

Your Azure logs may sometimes contain references to a user Liftr_Elastic. This user is created automatically by Azure as part of the integration with Elastic Cloud.

To check which of your Azure resources are currently being monitored, navigate to your Elasticsearch deployment and open the Monitored resources tab. Each resource shows one of the following status indicators:

  • Sending - Logs are currently being sent to the Elasticsearch cluster.
  • Logs not configured - Log collection is currently not configured for the resource. Open the Edit tags link to configure which logs are collected. For details about tagging resources, check Use tags to organize your Azure resources and management hierarchy in the Azure documentation.
  • N/A - Monitoring is not available for this resource type.
  • Limit reached - Azure resources can send diagnostic data to a maximum of five outputs. Data is not being sent to the Elasticsearch cluster because the output limit has already been reached.
  • Failed - Logs are configured but failed to ship to the Elasticsearch cluster. For help resolving this problem you can contact Support.
  • Region not supported - The Azure resource must be in one of the supported regions.
How do I ingest metrics from my Azure services?
Metrics are not supported as part of the current native integration. This will be implemented in a future phase. Metrics can still be collected from all Azure services using Metricbeat. For details, check Ingest other Azure metrics using the Metricbeat Azure module.
How can I monitor my Azure virtual machines in Elasticsearch?

You can monitor your Azure virtual machines by installing the Elastic Agent VM extension. Once enabled, the VM extension downloads the Elastic Agent, installs it, and enrols it to the Fleet Server. The Elastic Agent will then send system related logs and metrics to the Elastic Cloud cluster where you can find pre-built system dashboards showing the health and performance of your virtual machines.

A dashboard showing system metrics for the VM

Enabling and disabling VM extensions

To enable or disable a VM extension:

  1. In Azure, navigate to your Elasticsearch deployment.
  2. Select the Virtual machines tab
  3. Select one or more virtual machines
  4. Choose Install Extension or Uninstall Extension.
The Virtual Machines page in Azure

While it’s possible to enable or disable a VM extension directly from the VM itself, we recommend always enabling or disabling your Elasticsearch VM extensions from within the context of your Elasticsearch deployment.

Managing the Elastic Agent VM extension

Once installed on the virtual machine, you can manage Elastic Agent either from Fleet or locally on the host where it’s installed. We recommend managing the VM extension through Fleet, because it makes handling and upgrading the agents considerably easier. For more information on the Elastic Agent, check Manage your Elastic Agents.

Operating system compatibility matrix

The Azure Elastic Agent VM extension is supported on the following operating systems:

Platform Version

Windows

2008r2+

CentOS

6.10+

Debian

9,10

Oracle

6.8+

RHEL

7+

Ubuntu

16+

Troubleshootingedit

This section describes some scenarios that you may experience onboarding to Elastic Cloud through the Azure console. If you’re running into issues you can always get support.

I receive an error message about not having the required authorization.

When trying to access Elastic Cloud resources, you may get an error message indicating that the user must have the required authorization.

Error message displayed in the Elastic Cloud console: To access the resource {resource-name}

Elastic is not currently integrated with Azure user management, so sharing deployment resources through the Cloud console with other Azure users is not possible. However, sharing direct access to these resources is possible. For details, check Is the Elasticsearch native integration connected with Azure user management?.

My Elastic Cloud deployment creation failed.

When you attempt to create a new Elastic Cloud deployment, the deployment creation may proceed normally at first and then fail with a Your deployment failed error. In this case, the process results with a status message like the following:

{
  "code": "DeploymentFailed",
  "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
  "details": [
    {
      "code": "500",
      "message": "An error occurred during deployment creation. Please try again. If the problem persists, please contact support@elastic.co."
    }
  ]

One possible cause of a deployment creation failure is the default traffic filtering rules. A deployment will fail to create if you previously created a traffic filter and enabled the Include by default option. When this option is enabled, traffic to the deployment is blocked, including even traffic that is part of the native integration with Azure. As a result, some of the integration components are not successfully provisioned and the deployment creation fails.

Follow these steps to resolve the problem:

  1. Login to the Elasticsearch Service Console.
  2. Go to the Traffic filters page.
  3. Edit the traffic filter and disable the Include by default option.

    The Include by default option under Add to Deployments on the Traffic Filter page
  4. In Azure, create a new Elastic Cloud deployment.
  5. After the deployment has been created successfully, go back to the Traffic filters page in Elastic Cloud and re-enable the Include by default option.

If your deployment still does not create successfully, contact the Elastic Support Team for assistance.

I can’t SSO into my Elastic Cloud deployment.

When you try to access your Elastic Cloud deployment using single sign-on, the access may fail due to missing permission required by your Azure environment.

To resolve the problem, contact your Azure administrator and request SSO access to Elastic from your Azure account.

A UI dialog indicating that the Elasticsearch app in Azure requires Admin approval for SSO access.
My Azure Cloud Native integration logs are not being ingested.
When you set up monitoring for your Azure services, if your Azure and Elastic resources are in different subscriptions, you need to make sure that the Microsoft.Elastic resource provider is registered in the subscription in which the Azure resources exist. Check How do I monitor my existing Azure services? for details.

Getting supportedit

How do I get support?

Support is provided by Elastic. To open a support case:

  1. Navigate to the deployment overview page in the Azure console.
  2. Open the Support tab.
  3. Open the link to launch the Elastic console where you can provide further details.

    A member of the Elastic Support team will respond based on the SLA response time of your subscription.

    The New Support Request page in Azure

In case your Elastic Cloud resource is not fully set up and you’re not able to access the Support page, you can always send an email to support@elastic.co.

How can I change my subscription level / support level?
Your Elastic subscription level includes the support level. Check How can I change my Elastic Cloud subscription level? to make an update.