Azure Event Hub Input

Azure Event Hub input integration

What is an Elastic integration?
Version
1.13.0 (View all)
Compatible Kibana version(s)
8.13.0 or higher
Supported Serverless project types

What's this?
Security
Observability
Subscription level
What's this?
Basic

The Azure Event Hub Input integration is a generic integration that allows you to collect log categories from Azure services using Azure Event Hubs.

The azure-eventhub input uses the Event Processor Host. EPH can run across multiple processes and machines while load-balancing message consumers. More on this in the Azure event-hubs-go doc and Azure event-processor doc).

EPH state, such as leases on partitions and checkpoints in the event stream, is shared between receivers using an Azure Storage container. For this reason, users will have to create or use an existing storage account as a prerequisite to using this input.

Requirements and setup

Refer to the Azure Logs page for more information about setting up and using this integration.

Settings

eventhub : string It is a fully managed, real-time data ingestion service. Elastic recommends using only letters, numbers, and the hyphen (-) character for Event Hub names to maximize compatibility. You can use existing Event Hubs having underscores (_) in the Event Hub name; in this case, the integration will replace underscores with hyphens (-) when it uses the Event Hub name to create dependent Azure resources behind the scenes (e.g., the storage account container to store Event Hub consumer offsets). Elastic also recommends using a separate event hub for each log type as the field mappings of each log type differ.

consumer_group : string The publish/subscribe mechanism of Event Hubs is enabled through consumer groups. A consumer group is a view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets. Default value: $Default

connection_string : string The connection string required to communicate with Event Hubs, steps here https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string.

A Blob Storage account is required in order to store/retrieve/update the offset or state of the eventhub messages. This means that after stopping the filebeat azure module it can start back up at the spot that it stopped processing messages.

storage_account : string The name of the storage account the state/offsets will be stored and updated.

storage_account_key : string The storage account key, this key will be used to authorize access to data in your storage account.

storage_account_container : string The storage account container where the integration stores the checkpoint data for the consumer group. It is an advanced option to use with extreme care. You MUST use a dedicated storage account container for each Azure log type (activity, sign-in, audit logs, and others). DO NOT REUSE the same container name for more than one Azure log type. See Container Names for details on naming rules from Microsoft. The integration generates a default container name if not specified.

resource_manager_endpoint : string Optional, by default we are using the azure public environment, to override, users can provide a specific resource manager endpoint in order to use a different azure environment.

Resource manager endpoints:

# Azure ChinaCloud
https://management.chinacloudapi.cn/

# Azure GermanCloud
https://management.microsoftazure.de/

# Azure PublicCloud 
https://management.azure.com/

# Azure USGovernmentCloud
https://management.usgovcloudapi.net/

Logs

eventhub

The eventhub data stream of the Azure Logs package will collect any events that have been streamed through an azure event hub.

An example event for eventhub looks as following:

{
    "@timestamp": "2021-10-18T12:31:17.027Z",
    "agent": {
        "ephemeral_id": "42a183d5-df19-4008-8776-d9765bc01d50",
        "hostname": "docker-fleet-agent",
        "id": "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "7.15.0"
    },
    "azure-eventhub": {
        "consumer_group": "$Default",
        "enqueued_time": "2021-10-15T09:14:25.419Z",
        "eventhub": "insights-activity-logs",
        "offset": 274878093752,
        "sequence_number": 1215
    },
    "data_stream": {
        "dataset": "azure.eventhub",
        "namespace": "default",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
        "snapshot": true,
        "version": "7.15.0"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "azure.eventhub",
        "ingested": "2021-10-18T12:31:17Z",
        "kind": "event"
    },
    "host": {
        "architecture": "x86_64",
        "containerized": true,
        "hostname": "docker-fleet-agent",
        "id": "6505f7ca36739e7eb909bdb52bf3ec18",
        "ip": [
            "172.27.0.7"
        ],
        "mac": [
            "02-42-AC-14-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "Core",
            "family": "redhat",
            "kernel": "4.19.128-microsoft-standard",
            "name": "CentOS Linux",
            "platform": "centos",
            "type": "linux",
            "version": "7 (Core)"
        }
    },
    "input": {
        "type": "azure-eventhub"
    },
    "message": "{\\\"ReleaseVersion\\\":\\\"6.2021.39.11+d1f0e29.release_2021w39_az\\\",\\\"RoleLocation\\\":\\\"North Europe\\\",\\\"callerIpAddress\\\":\\\"77.170.179.229\\\",\\\"category\\\":\\\"Administrative\\\",\\\"correlationId\\\":\\\"a75a0752-ebbb-42bf-831f-43788a8c1193\\\",\\\"durationMs\\\":\\\"0\\\",\\\"identity\\\":{\\\"authorization\\\":{\\\"action\\\":\\\"Microsoft.ServiceBus\\/namespaces\\/delete\\\",\\\"evidence\\\":{\\\"principalId\\\":\\\"68b1adf93eb744b08eb8ce96522a08d3\\\",\\\"principalType\\\":\\\"User\\\",\\\"role\\\":\\\"Owner\\\",\\\"roleAssignmentId\\\":\\\"7f06f09dd6764b44930adbec3f10e92b\\\",\\\"roleAssignmentScope\\\":\\\"\\/providers\\/Microsoft.Management\\/managementGroups\\/5341238b-665c-4eb4-b259-b250371ae430\\\",\\\"roleDefinitionId\\\":\\\"8e3af657a8ff443ca75c2fe8c4bcb635\\\"},\\\"scope\\\":\\\"\\/subscriptions\\/7657426d-c4c3-44ac-88a2-3b2cd59e6dba\\/resourcegroups\\/obs-test\\/providers\\/Microsoft.ServiceBus\\/namespaces\\/testobs\\\"},\\\"claims\\\":{\\\"aio\\\":\\\"ATQAy\\/8TAAAAgFUjNWoJWKgHlAK2AL92UMeUsb6VD5zck\\/myDZPucX5V3Gc8SDMg5vTV28NUy5N7\\\",\\\"appid\\\":\\\"c44b4083-3bb0-49c1-b47d-974e53cbdf3c\\\",\\\"appidacr\\\":\\\"2\\\",\\\"aud\\\":\\\"https:\\/\\/management.core.windows.net\\/\\\",\\\"exp\\\":\\\"1634290798\\\",\\\"groups\\\":\\\"644c6686-9ef1-4b69-9410-107664a9e1f0,9ed1993c-ce9c-4915-a04d-58c6f5f7ee12,a953f548-26ab-47b2-be7d-65586b7bcc2a\\\",\\\"http:\\/\\/schemas.microsoft.com\\/2012\\/01\\/devicecontext\\/claims\\/identifier\\\":\\\"1060004c-63dc-465b-b868-ec6547176c58\\\",\\\"http:\\/\\/schemas.microsoft.com\\/claims\\/authnclassreference\\\":\\\"1\\\",\\\"http:\\/\\/schemas.microsoft.com\\/claims\\/authnmethodsreferences\\\":\\\"pwd,rsa\\\",\\\"http:\\/\\/schemas.microsoft.com\\/identity\\/claims\\/objectidentifier\\\":\\\"68b1adf9-3eb7-44b0-8eb8-ce96522a08d3\\\",\\\"http:\\/\\/schemas.microsoft.com\\/identity\\/claims\\/scope\\\":\\\"user_impersonation\\\",\\\"http:\\/\\/schemas.microsoft.com\\/identity\\/claims\\/tenantid\\\":\\\"4fa94b7d-a743-486f-abcc-6c276c44cf4b\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/givenname\\\":\\\"Mariana\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/name\\\":\\\"mariana@elastic.co\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/nameidentifier\\\":\\\"a9L2WR3XZN5ANzAqwLx_4aamU49JG6kqaE5JZkXdeNs\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/surname\\\":\\\"Dima\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/upn\\\":\\\"mariana@elastic.co\\\",\\\"iat\\\":\\\"1634286898\\\",\\\"ipaddr\\\":\\\"77.170.179.229\\\",\\\"iss\\\":\\\"https:\\/\\/sts.windows.net\\/4fa94b7d-a743-486f-abcc-6c276c44cf4b\\/\\\",\\\"name\\\":\\\"Mariana Dima\\\",\\\"nbf\\\":\\\"1634286898\\\",\\\"puid\\\":\\\"1003200045B17AD4\\\",\\\"rh\\\":\\\"0.AVEAfUupT0Onb0irzGwnbETPS4NAS8SwO8FJtH2XTlPL3zxRAA8.\\\",\\\"uti\\\":\\\"yUcYeZwj9EWeA-rTCtRwAA\\\",\\\"ver\\\":\\\"1.0\\\",\\\"wids\\\":\\\"5d6b6bb7-de71-4623-b4af-96380a352509\\\",\\\"xms_tcdt\\\":\\\"1469565974\\\"}},\\\"level\\\":\\\"Information\\\",\\\"operationName\\\":\\\"MICROSOFT.SERVICEBUS\\/NAMESPACES\\/DELETE\\\",\\\"properties\\\":{\\\"entity\\\":\\\"\\/subscriptions\\/7657426d-c4c3-44ac-88a2-3b2cd59e6dba\\/resourcegroups\\/obs-test\\/providers\\/Microsoft.ServiceBus\\/namespaces\\/testobs\\\",\\\"eventCategory\\\":\\\"Administrative\\\",\\\"hierarchy\\\":\\\"4fa94b7d-a743-486f-abcc-6c276c44cf4b\\/5341238b-665c-4eb4-b259-b250371ae430\\/7657426d-c4c3-44ac-88a2-3b2cd59e6dba\\\",\\\"message\\\":\\\"Microsoft.ServiceBus\\/namespaces\\/delete\\\"},\\\"resourceId\\\":\\\"\\/SUBSCRIPTIONS\\/7657426D-C4C3-44AC-88A2-3B2CD59E6DBA\\/RESOURCEGROUPS\\/OBS-TEST\\/PROVIDERS\\/MICROSOFT.SERVICEBUS\\/NAMESPACES\\/TESTOBS\\\",\\\"resultSignature\\\":\\\"Started.\\\",\\\"resultType\\\":\\\"Start\\\",\\\"tenantId\\\":\\\"4fa94b7d-a743-486f-abcc-6c276c44cf4b\\\",\\\"time\\\":\\\"2021-10-15T09:08:29.9268177Z\\\"}\\r\\n",
    "tags": [
        "azure-eventhub"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
azure-eventhub.consumer_group
consumer group
keyword
azure-eventhub.enqueued_time
The enqueued time.
keyword
azure-eventhub.eventhub
Event hub name.
keyword
azure-eventhub.offset
Offset
long
azure-eventhub.partition_id
Partition ID
keyword
azure-eventhub.sequence_number
Sequence number
long
azure.correlation_id
Correlation ID
keyword
azure.resource.authorization_rule
Authorization rule
keyword
azure.resource.group
Resource group
keyword
azure.resource.id
Resource ID
keyword
azure.resource.name
Name
keyword
azure.resource.namespace
Resource type/namespace
keyword
azure.resource.provider
Resource type/namespace
keyword
azure.subscription_id
Azure subscription ID
keyword
azure.tenant_id
tenant ID
keyword
cloud.image.id
Image ID for the cloud instance.
keyword
data_stream.dataset
Data stream dataset name.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.module
Event module
constant_keyword
geo.city_name
City name.
keyword
geo.continent_name
Name of the continent.
keyword
geo.country_iso_code
Country ISO code.
keyword
geo.country_name
Country name.
keyword
geo.location
Longitude and latitude.
geo_point
host.containerized
If the host is a container.
boolean
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
input.type
Input type
keyword

Changelog

VersionDetailsKibana version(s)

1.13.0

Enhancement View pull request
Add structured log categories to Azure Firewall.

8.13.0 or higher

1.12.0

Enhancement View pull request
ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

Breaking change View pull request
Updated "event.outcome" values from "Succeeded" to "success" and "Failed" to "failure".

8.13.0 or higher

1.11.4

Bug fix View pull request
Replace Azure AD with Microsoft Entra ID.

8.12.0 or higher

1.11.3

Bug fix View pull request
Update Azure Audit Logs pipeline with support for initiated_by user fields.

8.12.0 or higher

1.11.2

Bug fix View pull request
Add missing ECS field definitions.

8.12.0 or higher

1.11.1

Enhancement View pull request
Update description for event hub parameter name recommendations.

8.12.0 or higher

1.11.0

Enhancement View pull request
Use ecs definition of the 'event.dataset' field for eventhub

8.12.0 or higher

1.10.0

Enhancement View pull request
Add Microsoft Graph Activity Logs

8.12.0 or higher

1.9.2

Enhancement View pull request
Add docs for running the integration behind a firewall.

8.12.0 or higher

1.9.1

Bug fix View pull request
Set field type to password on secret fields.

8.12.0 or higher

1.9.0

Enhancement View pull request
Add support for integration secrets

8.12.0 or higher

1.8.3

Enhancement View pull request
Add caller_ip_address field in pipeline for Azure sign-in logs.

8.8.0 or higher

1.8.2

Enhancement View pull request
Update Azure logs documentation.

8.8.0 or higher

1.8.1

Enhancement View pull request
Update AD Logs documentation.

8.8.0 or higher

1.8.0

Enhancement View pull request
Allow rerouting of Azure logs events to a different data stream.

8.8.0 or higher

1.7.0

Enhancement View pull request
Rebrand Azure Spring Cloud Logs to Azure Spring Apps

8.6.0 or higher

1.6.0

Enhancement View pull request
Update the package format_version to 3.0.0.

8.6.0 or higher

1.5.33

Bug fix View pull request
Handle json.properties.clientIp as an alias of json.properties.clientIP in application gateway logs

8.6.0 or higher

1.5.32

Bug fix View pull request
Fix mappings for azure.activitylogs.claims.*.

8.6.0 or higher

1.5.31

Enhancement View pull request
Migration of Azure AD Identity Protection dashboard to Lens.

8.6.0 or higher

1.5.30

Enhancement View pull request
Migration of Azure AD Provisioning Logs dashboard to Lens.

8.6.0 or higher

1.5.29

Enhancement View pull request
Fix Azure dashboards descriptions and titles.

8.6.0 or higher

1.5.28

Enhancement View pull request
Migration Alerts Overview dashboard to Lens

8.6.0 or higher

1.5.27

Bug fix View pull request
Fix in Firewall dashboards

8.6.0 or higher

1.5.26

Bug fix View pull request
Handle duplicate user_agent.original field in signinlogs logs

8.6.0 or higher

1.5.25

Bug fix View pull request
Handle duplicate url.path field in application gateway logs

8.6.0 or higher

1.5.24

Bug fix View pull request
Handle firewall events for DNAT'ed requests with attributes

8.6.0 or higher

1.5.23

Enhancement View pull request
Update Azure Logs screenshot

8.6.0 or higher

1.5.22

Enhancement View pull request
Migration of Azure Cloud Overview dashboard to Lens and style changes

8.6.0 or higher

1.5.21

Enhancement View pull request
Migration of User Activity Dashboard to Lens

8.6.0 or higher

1.5.20

Enhancement View pull request
Integration settings UI for sanitization

8.6.0 or higher

1.5.17

Enhancement View pull request
Migration of Spring Cloud Overview dashboard to Lens

8.6.0 or higher

1.5.16

Enhancement View pull request
Migration of Azure Spring Cloud Logs Application Cloud Logs dashboard to Lens

8.6.0 or higher

1.5.15

Enhancement View pull request
Migration of Spring Cloud System Logs dashboard to Lens

8.6.0 or higher

1.5.14

Enhancement View pull request
Enhancement/Improving performance of the dashboards

8.6.0 or higher

1.5.13

Enhancement View pull request
Extend the Storage Account container documentation and add link to requiements and setup instructions

7.16.0 or higher
8.0.0 or higher

1.5.12

Enhancement View pull request
Added categories and/or subcategories.

7.16.0 or higher
8.0.0 or higher

1.5.11

Enhancement View pull request
Add a new message format to the AzureFirewallNetworkRule log category

7.16.0 or higher
8.0.0 or higher

1.5.10

Bug fix View pull request
Check for 'event.original' already existing in Application Gateway and Event Hub ingest pipelines

7.16.0 or higher
8.0.0 or higher

1.5.9

Bug fix View pull request
Check for 'event.original' already existing in firewall logs ingest pipeline

7.16.0 or higher
8.0.0 or higher

1.5.8

Bug fix View pull request
Add storage_account_container option to the Application Gateway integration

7.16.0 or higher
8.0.0 or higher

1.5.7

Bug fix View pull request
Fix parsing of authentication_processing_details field in signin logs

7.16.0 or higher
8.0.0 or higher

1.5.6

Bug fix View pull request
Fix parsing error client port is blank and adjust for timeStamp

7.16.0 or higher
8.0.0 or higher

1.5.5

Bug fix View pull request
Rename identity as identity_name when the value is a string

7.16.0 or higher
8.0.0 or higher

1.5.4

Enhancement View pull request
Enable Event Hub integration by default and improve documentation

7.16.0 or higher
8.0.0 or higher

1.5.3

Enhancement View pull request
Data streams start as disabled on new installs

7.16.0 or higher
8.0.0 or higher

1.5.2

Bug fix View pull request
Fix PR link in changelog

7.16.0 or higher
8.0.0 or higher

1.5.1

Bug fix View pull request
Fix documentations formatting (remove extra 'Overview' heading)

7.16.0 or higher
8.0.0 or higher

1.5.0

Enhancement View pull request
Add Azure Application Gatewaty data stream

7.16.0 or higher
8.0.0 or higher

1.4.1

Enhancement View pull request
Update Azure Logs documentation

7.16.0 or higher
8.0.0 or higher

1.4.0

Enhancement View pull request
Add two new data streams to the Azure AD logs integration: Azure Identity Protection logs and Provisioning logs

7.16.0 or higher
8.0.0 or higher

1.3.0

Enhancement View pull request
Add the possibility to override the default generated storage account container

7.16.0 or higher
8.0.0 or higher

1.2.3

Enhancement View pull request
Update docs with recommended Event Hub configuration

7.16.0 or higher
8.0.0 or higher

1.2.2

Enhancement View pull request
Update package name and description to align with standard wording

7.16.0 or higher
8.0.0 or higher

1.2.1

Bug fix View pull request
Fix Azure Sign-in logs ingest pipeline bug

7.16.0 or higher
8.0.0 or higher

1.2.0

Enhancement View pull request
Support Azure firewall logs

—

1.1.11

Bug fix View pull request
Improve support for event.original field from upstream forwarders.

—

1.1.10

Enhancement View pull request
Update readme with links to Microsoft documentation

7.16.0 or higher
8.0.0 or higher

1.1.9

Bug fix View pull request
Improve handling of IPv6 IP addresses.

—

1.1.8

Enhancement View pull request
Update docs with details about Event Hub name recommendations

7.16.0 or higher
8.0.0 or higher

1.1.7

Bug fix View pull request
Add geo.name and result_description fields in platformlogs

7.16.0 or higher
8.0.0 or higher

1.1.6

Bug fix View pull request
Fix azure.activitylogs.identity with a a concrete value

Bug fix View pull request
Add identity_name, tenant_id, level and operation_version into activity logs

7.16.0 or higher
8.0.0 or higher

1.1.5

Enhancement View pull request
Add documentation for multi-fields

—

1.1.4

Bug fix View pull request
Fix event.duration field mapping conflict in all Azure data streams.

—

1.1.3

Enhancement View pull request
Added the forwarded tag by default to all log types.

—

1.1.2

Bug fix View pull request
Add device_detail.is_compliant and device_detail.is_managed fields

Bug fix View pull request
Change authentication_requirement_policies to flattened type

7.16.0 or higher
8.0.0 or higher

1.1.1

Bug fix View pull request
Fix field mapping conflict in the auditlogs data stream for client.ip. Changed azure-eventhub.offset and azure-eventhub.sequence_number to longs from keyword in the eventhub data stream.

—

1.1.0

Enhancement View pull request
Support new Azure audit logs and signin logs

—

1.0.1

Enhancement View pull request
Remove beta release tag from data streams

7.16.0 or higher
8.0.0 or higher

1.0.0

Enhancement View pull request
Move azure package to GA

7.16.0 or higher
8.0.0 or higher

0.12.3

Enhancement View pull request
Update to ECS 8.0

—

0.12.2

Bug fix View pull request
Regenerate test files using the new GeoIP database

—

0.12.1

Bug fix View pull request
Change test public IPs to the supported subset

—

0.12.0

Enhancement View pull request
Release azure package for v8.0.0

—

0.11.0

Enhancement View pull request
Add Azure Event Hub Input

—

0.10.1

Enhancement View pull request
Uniform with guidelines

—

0.10.0

Enhancement View pull request
signinlogs - Add support for ManagedIdentitySignInLogs, NonInteractiveUserSignInLogs, and ServicePrincipalSignInLogs.

—

0.9.2

Bug fix View pull request
Prevent pipeline script error

—

0.9.1

Bug fix View pull request
Fix logic that checks for the 'forwarded' tag

—

0.9.0

Enhancement View pull request
Update to ECS 1.12.0

—

0.8.6

Bug fix View pull request
Add ECS client.ip mapping

—

0.8.5

Enhancement View pull request
Update docs and logo

—

0.8.4

Enhancement View pull request
Convert to generated ECS fields

—

0.8.3

Enhancement View pull request
Import geo_points from ECS

—

0.8.2

Enhancement View pull request
Update error message

—

0.8.1

Enhancement View pull request
Add support for springcloud logs inside the platformlogs pipeline

—

0.8.0

Enhancement View pull request
Import ECS field definitions

—

0.7.0

Enhancement View pull request
Add spring cloud logs

—

0.6.2

Enhancement View pull request
update to ECS 1.11.0

—

0.6.1

Enhancement View pull request
Escape special characters in docs

—

0.6.0

Enhancement View pull request
Update integration description

—

0.5.1

Enhancement View pull request
Re-add pipeline changes for invalid json

—

0.5.0

Enhancement View pull request
Add input groups

—

0.4.0

Enhancement View pull request
Set "event.module" and "event.dataset"

—

0.3.1

Enhancement View pull request
sync package with module changes

—

0.3.0

Enhancement View pull request
update to ECS 1.10.0 and adding event.original options

—

0.2.3

Enhancement View pull request
update to ECS 1.9.0

—

0.2.2

Bug fix View pull request
Correct sample event file.

—

0.2.1

Bug fix View pull request
Add check for empty configuration options.

—

0.2.0

Enhancement View pull request
Add changes to use ECS 1.8 fields.

—

0.0.1

Enhancement View pull request
initial release

—

On this page