Migrate Logstash Pipelines from Azure Event Hubs to Kafka Input Plugin

Introduction

Azure Event Hubs natively supports the Apache Kafka protocol, which means you no longer need the logstash-input-azure_event_hubs plugin or an external Blob Storage account for offset checkpointing. Switching to logstash-input-kafka removes that storage dependency, reduces costs, and delivers up to 2.5x higher throughput.

This guide walks you through the migration: why it matters, how to convert your existing configuration, parameter mapping between the two plugins, and how to adapt proxy setups.

Why migrate?

The migration from the Azure Event Hubs plugin to the Kafka input plugin is motivated by several factors:

1. Azure Event Hubs already speaks Kafka natively. Event Hubs exposes a built-in Apache Kafka endpoint on Standard, Premium, and Dedicated tiers. This means the logstash-input-azure_event_hubs plugin is no longer necessary. The standard logstash-integration-kafka (input) plugin connects directly to the same service with no extra Azure-side configuration.

2. No more Blob Storage for offset checkpointing. The AMQP-based plugin requires an external Azure Blob Storage account to track consumer offsets. This means provisioning and maintaining a storage account, plus paying for every checkpoint write. With the Kafka protocol, offset tracking is handled internally by Azure Event Hubs at no extra cost, removing the need for external storage.

3. GPv1 storage retirement is coming, and GPv2 costs more. Microsoft will retire general-purpose v1 storage accounts in October 2026. Accounts not manually upgraded to GPv2 by then will be migrated automatically. The logstash-input-azure_event_hubs plugin works correctly with GPv2, so existing pipelines will not break. However, GPv2 can bring higher transactional costs, especially for checkpoint-heavy workloads. By switching to the Kafka input plugin, this concern is eliminated: no storage account means nothing to upgrade and nothing to pay for.

   Not ready to migrate yet? Reducing GPv2 costs in the meantime is possible. GPv2 transaction pricing is significantly more expensive than GPv1's flat rate. Increasing the checkpoint_interval setting above its default of 5 seconds reduces write operations and lowers the cost impact. The cost difference can be estimated using the Azure Pricing Calculator.

   Example for East US and Local Retention Storage. Write operation cost comparison (per 10,000 write operations):

   • GPv1 (flat): $0.00036

   • GPv2 (Hot tier): $0.050

   That's roughly a 140x increase in write operation costs.

4. Broader community and active maintenance. The Kafka input plugin is more widely used across Logstash deployments and receives regular updates aligned with the Kafka ecosystem. Moving to it reduces long-term operational risk and keeps your pipeline on a well-supported path.

5. Better throughput. The Kafka input plugin consistently outperforms the Azure Event Hubs plugin when consuming from the same namespace. See the Performance Comparison section for measured results.

Requirements to enable the Kafka interface

The Kafka interface is built into Azure Event Hubs. You don't need to enable or configure anything in the Azure portal.

The only requirement is that your Event Hubs namespace is on the Standard, Premium, or Dedicated tier. The Basic tier does not support the Kafka protocol.

See the Tiers comparison table for details.

Converting your configuration

This section walks through converting an existing logstash-input-azure_event_hubs configuration to logstash-input-kafka, starting with the simplest single-hub scenario and building up to multi-hub and advanced use cases.

Key behavior changes

Before changing any configuration, be aware of two important differences:

1. No more Blob Storage for offsets. The Kafka input plugin tracks offsets internally through the Azure Event Hubs service at no extra cost. The storage_connection and storage_container parameters have no equivalent. There is nothing to provision, maintain, or pay for.

2. Consumer offsets don't carry over. AMQP consumer groups and Kafka consumer groups are completely separate, even if they share the same name. When the Kafka input plugin connects for the first time, Azure auto-creates the Kafka consumer group specified in group_id (default: logstash). It will not read the old Blob Storage checkpoints or resume from where the legacy plugin left off. It starts fresh.

Limit: A maximum of 1,000 simultaneous Kafka consumer groups per namespace is allowed. See the Event Hubs vs. Kafka Consumer Groups FAQ.

Authentication

The logstash-input-azure_event_hubs plugin only supports SAS (Shared Access Signature) authentication via connection strings. The same SAS credentials work with the Kafka plugin through SASL PLAIN, as shown in the Single Event Hub (basic migration) example below.

Single Event Hub (basic migration)

Most pipelines start with a single Event Hub, SAS authentication, and Blob Storage checkpointing. The following example shows the baseline azure_event_hubs configuration and its direct Kafka equivalent.

Before (legacy Azure Event Hubs input):

input {
  azure_event_hubs {
    event_hub_connections => ["Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY>;EntityPath=<EVENT_HUB_NAME>"]
    storage_connection => "DefaultEndpointsProtocol=https;AccountName=<STORAGE_ACCOUNT_NAME>;AccountKey=<STORAGE_ACCOUNT_KEY>;EndpointSuffix=core.windows.net"
    consumer_group => "<CONSUMER_GROUP_NAME>"
    storage_container => "<STORAGE_NAME>"
  }
}

After (Kafka input):

input {
  kafka {
    # The Namespace name and the mandatory Kafka SSL port
    bootstrap_servers => "<NAMESPACE>.servicebus.windows.net:9093"
    
    topics => ["<EVENT_HUB_NAME>"]
    group_id => "<KAFKA_CONSUMER_GROUP_NAME>"
    security_protocol => "SASL_SSL"
    sasl_mechanism => "PLAIN"
    
    # Need to create a 'jaas.conf' file storing Username and Password (username is always '$ConnectionString')
    jaas_path => "path/to/jaas.conf"
  }
}

KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="$ConnectionString" 
    password="Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY>";
};

# Inline JAAS configuration (substitutes jaas_path)
    sasl_jaas_config => "org.apache.kafka.common.security.plain.PlainLoginModule required username='$ConnectionString' password='Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY>';"

Multiple Event Hubs with a single Kafka input

If your SAS policy has namespace-level read rights (not just a single Event Hub), you can consume from multiple Event Hubs with a single kafka input by listing multiple topics:

input {
  kafka {
    bootstrap_servers => "<NAMESPACE>.servicebus.windows.net:9093"
    topics => ["<EVENT_HUB_1>", "<EVENT_HUB_2>", "<EVENT_HUB_3>"]
    group_id => "<KAFKA_CONSUMER_GROUP_NAME>"
    security_protocol => "SASL_SSL"
    sasl_mechanism => "PLAIN"
    jaas_path => "path/to/jaas.conf"
  }
}

Configuration parameters mapping

The following section maps each logstash-input-azure_event_hubs parameter to its logstash-input-kafka equivalent, with usage notes and example configurations.

1. config_mode: No direct equivalent. Kafka doesn't have "basic" vs "advanced" modes. To consume from multiple hubs with different settings, define multiple kafka {} input blocks or list multiple topics. The basic mode conversion is covered in Single Event Hub (basic migration).

   Here is an advanced-mode example with two Event Hubs in the same namespace:

   input {
       azure_event_hubs {
           config_mode => "advanced"
           storage_connection => "DefaultEndpointsProtocol=https;AccountName=<STORAGE_ACCOUNT>;..."
           event_hubs => [
               {"<EVENT_HUB_1>" => {
                   event_hub_connection => "Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<KEY_1>;SharedAccessKey=<ACCESS_KEY>;EntityPath=<EVENT_HUB_1>"
                   consumer_group => "<CONSUMER_GROUP_1>"
               }},
               {"<EVENT_HUB_2>" => {
                   event_hub_connection => "Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<KEY_2>;SharedAccessKey=<ACCESS_KEY>;EntityPath=<EVENT_HUB_2>"
                   consumer_group => "<CONSUMER_GROUP_2>"
               }}
           ]
       }
   }
   

   input {
       kafka {
           bootstrap_servers => "<NAMESPACE>.servicebus.windows.net:9093"
           topics => ["<EVENT_HUB_1>"]
           group_id => "<KAFKA_CONSUMER_GROUP_1>"
           security_protocol => "SASL_SSL"
           sasl_mechanism => "PLAIN"
           sasl_jaas_config => "...<KEY_1>..."
       }
       kafka {
           bootstrap_servers => "<NAMESPACE>.servicebus.windows.net:9093"
           topics => ["<EVENT_HUB_2>"]
           group_id => "<KAFKA_CONSUMER_GROUP_2>"
           security_protocol => "SASL_SSL"
           sasl_mechanism => "PLAIN"
           sasl_jaas_config => "...<KEY_2>..."
       }
   }
   

2. checkpoint_interval: This corresponds to auto_commit_interval_ms.

   In the Azure plugin, this controls how often a write operation hits the Blob Storage container to save the reading offset. In the Kafka plugin, it controls how often the consumer commits its offset to the Event Hubs service.

   Note Keep enable_auto_commit set to true (default) while configuring auto_commit_interval_ms parameter.

   Azure config:

   input {
       azure_event_hubs {
           # ... other params ...
           checkpoint_interval => 10 # in seconds
       }
   }
   

   Kafka equivalent:

   input {
       kafka {
           # ... other params ...
           auto_commit_interval_ms => 10000 # in milliseconds 
       }
   }
   

3. decorate_events: This parameter exists in both plugins with the same name and behavior.

4. initial_position: This corresponds to auto_offset_reset.

   Both parameters control where to start reading when no prior offset is found at checkpoint storage. Options differ slightly:

   • Azure: beginning, end, look_back

   • Kafka: earliest, latest, by_duration:<duration>, none

   The difference between beginning-end and earliest-latest is purely terminology.

   Azure config:

   input {
       azure_event_hubs {
           initial_position => "beginning"
       }
   }
   

   Kafka equivalent:

   input {
       kafka {
           auto_offset_reset => "earliest"
       }
   }
   

   Azure Value	Kafka Value	Notes
   beginning	earliest
   end	latest
   look_back	by_duration:<duration>	Duration in ISO 8601 format (e.g., by_duration:PT1H for 1 hour). Requires logstash-integration-kafka 12.1.0+.

   The by_duration option was introduced in Apache Kafka client 4.0.0 and is available in logstash-integration-kafka version 12.1.0 and later. The version bundled with the latest Logstash release is older than 12.1.0, so a manual gem update is needed:

   <LOGSTASH_HOME>/bin/logstash-plugin install --version 12.1.0 logstash-integration-kafka
   

   Replace <LOGSTASH_HOME> with the Logstash installation directory (e.g., /usr/share/logstash for DEB/RPM packages).

   Note: Since Kafka can't read the old Blob Storage checkpoints, it treats the migration as a first-time connection. To avoid reprocessing data the legacy plugin already handled, set auto_offset_reset => "latest" for the initial deployment.

5. max_batch_size: This corresponds to max_poll_records.

   Both parameters define the maximum number of messages to fetch in a single poll/batch operation.

   Azure config:

   input {
       azure_event_hubs {
           max_batch_size => 125
       }
   }
   

   Kafka equivalent:

   input {
       kafka {
           max_poll_records => "125"
       }
   }
   

6. threads: This corresponds to consumer_threads.

   Both parameters control the number of threads used to consume messages concurrently. In Azure, the minimum is 2 (with 1 Event Hub + 1), while in Kafka the default is 1 thread.

   Azure config:

   input {
       azure_event_hubs {
           threads => 8
       }
   }
   

   Kafka equivalent:

   input {
       kafka {
           consumer_threads => 8
       }
   }
   

Performance Comparison

We tested both plugins under identical conditions: same Logstash instance, same Event Hub namespace, same number of partitions, and same batch/thread configuration. The absolute numbers are environment-specific, but the relative difference is what matters.

Across all payload sizes, the Kafka input plugin delivers 1.7x to 2.5x higher throughput. The gain is most noticeable with small payloads, where protocol overhead dominates. Beyond the infrastructure simplification (no Blob Storage, no GPv2 concerns), you also get a clear performance win.

Proxy connection configuration

If the Logstash instance connects directly to Azure Event Hubs without a proxy, this section can be skipped.

Proxy setups require special attention during this migration because the two plugins use fundamentally different protocols.

Azure Event Hubs plugin setup (reference)

The logstash-input-azure_event_hubs plugin supports HTTPS proxies. The setup involves:

1. Set the proxy environment variable:

   export https_proxy="https://my_proxy:8080"
   

2. Add the WebSockets transport flag to the Event Hubs connection string:

   ;TransportType=AmqpWebSockets
   

3. Add the following JVM options (Logstash jvm.options):

   -Dhttp.proxyHost=my_proxy
   -Dhttp.proxyPort=8080
   -Dhttps.proxyHost=my_proxy
   -Dhttps.proxyPort=8443
   -Dhttp.nonProxyHosts=localhost|127.0.0.1
   

Migrating to a TCP (Layer 4) proxy

The proxy setup from the Azure plugin is not compatible with the Kafka client. The Azure plugin communicates over AMQP/WebSockets (HTTP layer), which is why JVM proxy settings and TransportType=AmqpWebSockets work. The Kafka plugin opens a raw TCP socket to the broker. It never makes an HTTP request, so JVM HTTP proxy settings are ignored entirely. If the environment requires a proxy, the HTTP proxy needs to be replaced with a TCP (Layer 4) proxy.

Step 1: Configure /etc/hosts

The Kafka client verifies that the TLS certificate matches the hostname in bootstrap_servers. Since the certificate is issued for *.servicebus.windows.net, bootstrap_servers must use the real Event Hubs FQDN, not the proxy address. A DNS override routes the FQDN to the proxy IP:

# /etc/hosts
<PROXY_HOST_IP>  <NAMESPACE>.servicebus.windows.net

Step 2: Logstash configuration

The Logstash configuration is identical to a non-proxied setup. The /etc/hosts override transparently routes traffic through the proxy, so bootstrap_servers still uses the Event Hubs FQDN:

input {
  kafka {
    bootstrap_servers => "<NAMESPACE>.servicebus.windows.net:9093"
    topics => ["<EVENT_HUB_NAME>"]
    security_protocol => "SASL_SSL"
    sasl_mechanism => "PLAIN"
    group_id => "<GROUP_ID>"
    jaas_path => "<PATH_TO_JAAS_FILE>"
  }
}

If the TCP proxy runs on the same host as Logstash or within a trusted network segment, the DNS override is not needed. Instead, point bootstrap_servers directly to the proxy IP (e.g., <PROXY_HOST_IP>:9093) and change security_protocol to SASL_PLAINTEXT. This delegates the TLS handshake to the proxy, while the link between Logstash and the proxy stays unencrypted. Only use this configuration when the Logstash-to-proxy path is secure.

input {
  kafka {
    bootstrap_servers => "<PROXY_HOST_IP>:9093"
    security_protocol => "SASL_PLAINTEXT"
  }
}

Frequently asked questions

Are events lost when switching from the Azure Event Hubs plugin to the Kafka plugin?

No. Events remain available within the configured retention period regardless of which protocol reads them. What changes is where the consumer starts reading. Since the Kafka plugin cannot access the AMQP plugin's Blob Storage checkpoints, it starts from scratch. Set auto_offset_reset => "earliest" to reprocess all retained events, or auto_offset_reset => "latest" to consume only new ones from the switchover point. See the initial_position mapping for details.

What happens to the Azure Blob Storage account after migration?

It is no longer needed for offset checkpointing. Once the Kafka plugin is confirmed to be consuming correctly and the azure_event_hubs input has been decommissioned, the storage account (or at least the checkpoint container) can be safely deleted. If the storage account is used for other purposes, only remove the specific container referenced in storage_container.

Can the same consumer group name be reused?

Yes, but it has no practical effect. AMQP and Kafka consumer groups are completely independent even if they share the same name. They use different protocols, different offset storage, and different scoping rules. Reusing the name will not cause the Kafka plugin to resume from the AMQP plugin's last checkpoint.

Are other authentication methods supported?

The logstash-input-azure_event_hubs plugin only supports SAS connection strings, so SAS is the only credential that needs to be carried over. There is no Entra ID, OAUTHBEARER, or managed identity configuration to migrate. The logstash-input-kafka plugin does support SASL OAUTHBEARER, so adopting token-based authentication becomes possible after migration.

What if the proxy only allows traffic on port 443?

The Kafka endpoint on Azure Event Hubs requires port 9093. If the TCP proxy only forwards port 443, it must be reconfigured to also allow port 9093 for the Event Hubs FQDN (*.servicebus.windows.net). Azure Event Hubs does not expose a Kafka listener on port 443.

Next steps

With the GPv1 retirement deadline (October 2026) approaching, starting this migration sooner reduces the time spent managing storage infrastructure that is no longer needed.

If any issues arise during migration:

• Usage questions or help with configuration: Post on the Elastic Discuss forum.

• Bugs or unexpected behavior in the Kafka plugin: Open an issue in the logstash-integration-kafka.

Related resources

• Kafka input plugin documentation: Full reference for all logstash-input-kafka configuration parameters.
• Azure Event Hubs input plugin documentation: Full reference for the legacy plugin being replaced.
• Azure Event Hubs for Apache Kafka overview: Microsoft's documentation on the built-in Kafka endpoint in Event Hubs.
• Event Hubs quotas and tier comparison: Tier requirements for Kafka protocol support.