Overview

#

Use kafka cli tooling to work with Event Hubs. This same approach can be used to configure UI’s (such as kafbat ) and other tooling to use Event Hubs.

Apache Kafka and Azure Event Hubs conceptual mapping

#

This table is from Apache Kafka and Azure Event Hubs conceptual mapping

Setup

#

Example from Azure Docs Shared Access Signature (SAS)

We use the connection string access which can be retrieved from the Azure Portal UI.

Event Hubs -> Choose an Environment -> Pick an Event Hub -> Shared access policies -> Pick a policy -> Copy the connection string

bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";

As an example, this would be similar to a real output where mynamespace is the actual Azure namespace.

The thing to note is that username is literally set to "$ConnectionString". All of the credentials are in the password field. Event Hubs expects to see the magic, literal string "$ConnectionString" for username.

# myconf.conf
bootstrap.servers=mynamespace.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";

Let’s get ready to use it

export EH_CONFIG=myconf.conf
export EH_BOOTSTRAP_URL="mynamespace.servicebus.windows.net:9093"

List Topics

#

# List Topics
kafka-topics \
--list \
--bootstrap-server $EH_BOOTSTRAP_URL \
--command-config $EH_CONFIG

Describe a specific topic

#

# Describe a specific topic
export TARGET_TOPIC="my.topic.name"

kafka-topics \
--describe \
--topic $TARGET_TOPIC \
--bootstrap-server $EH_BOOTSTRAP_URL \
--command-config $EH_CONFIG

List configs for a specific topic

#

# List configs for a specific topic
export TARGET_TOPIC="my.topic.name"

kafka-configs \
--describe \
--entity-type topics \
--entity-name $TARGET_TOPIC \
--bootstrap-server $EH_BOOTSTRAP_URL \
--command-config $EH_CONFIG \
--all

List configs for all topics

#

# List configs for all topics
kafka-configs \
--describe \
--entity-type topics \
--bootstrap-server $EH_BOOTSTRAP_URL \
--command-config $EH_CONFIG \
--all &>> topicConfigs.log