Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Azure Event Hubs Source Connector for Confluent Platform¶

Note

If you are using Confluent Cloud, see https://docs.confluent.io/cloud/current/connectors/cc-azure-event-hubs-source.html for the cloud Quick Start.

The Kafka Connect Azure Event Hubs Source Connector is used to poll data from Azure Event Hubs and persist the data to a Apache Kafka® topic.

The Event Hubs EventData record has System Property and custom User Property map fields. System properties are set by Event Hubs and custom user properties can include string data that is useful for downstream processing (sender ID, message importance, and so on). These properties are added to the Kafka record header as maps of string keys to string values. When the records are transported out of Confluent Platform, these properties can be reconstructed for downstream applications.

Install the Azure Event Hubs Connector¶

You can install this connector by using the Confluent Hub client (recommended) or you can manually download the ZIP file.

Install the connector using Confluent Hub¶

Prerequisite: Confluent Hub Client must be installed. This is installed by default with Confluent Enterprise.

Navigate to your Confluent Platform installation directory and run the following command to install the latest (latest) connector version. The connector must be installed on every machine where Connect will run.

confluent-hub install confluentinc/kafka-connect-azure-event-hubs:latest

You can install a specific version by replacing latest with a version number. For example:

confluent-hub install confluentinc/kafka-connect-azure-event-hubs:1.1.4

Install the connector manually¶

Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.

License¶

You can use this connector for a 30-day trial period without a license key.

After 30 days, this connector is available under a Confluent enterprise license. Confluent issues enterprise license keys to subscribers, along with providing enterprise-level support for Confluent Platform and your connectors. If you are a subscriber, please contact Confluent Support at support@confluent.io for more information.

See Confluent Platform license for license properties and License topic configuration for information about the license topic.

Configuration Properties¶

For a complete list of configuration properties for this connector, see Azure Event Hubs Source Connector Configuration Properties.

Usage Notes¶

Azure Event Hubs use Shared access policies for authentication and authorization. For more information, see the Event Hubs Authentication and Security Model Overview.

REST-based example¶

This configuration is used typically along with distributed workers. Write the following JSON to connector.json, configure all of the required values, and use the command below to post the configuration to one the distributed connect workers. Refer to REST API for more information about the Kafka Connect.

Connect Distributed REST example:

 {
   "name": "EventHubsSourceConnector1",
   "config": {
     "confluent.topic.bootstrap.servers": "< Required Configuration >",
     "connector.class": "io.confluent.connect.azure.eventhubs.EventHubsSourceConnector",
     "kafka.topic": "< Required Configuration >",
     "tasks.max": "1",
     "max.events": "< Optional Configuration >",
     "azure.eventhubs.sas.keyname": "< Required Configuration >",
     "azure.eventhubs.sas.key": "< Required Configuration >",
     "azure.eventhubs.namespace": "< Required Configuration >",
     "azure.eventhubs.hub.name": "< Required Configuration >"
   }
 }

Use curl to post the configuration to one of the Kafka Connect Workers. Change http://localhost:8083/ to the endpoint of one of your Kafka Connect workers.

Create a new connector:

curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors

Update an existing connector:

curl -s -X PUT -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors/EventHubsSourceConnector1/config

Quick Start¶

The Azure Event Hubs source connector is used to poll data from an Event Hub, and write into a Kafka topic. Before you begin, you need to have an Azure subscription with the privilege to create resource group and service. Using the Azure portal, create a namespace and event hub. Then produce some events to the hub using Event Hubs API.

Note

For an example of how to get Kafka Connect connected to Confluent Cloud, see Distributed Cluster in Connect Kafka Connect to Confluent Cloud.

Preliminary Setup¶

Prerequisites

Confluent Platform
Confluent CLI (requires separate installation)

Navigate to your Confluent Platform installation directory and run the following command to install the latest connector version:

confluent-hub install confluentinc/kafka-connect-azure-event-hubs:latest

You can install a specific version by replacing latest with a version number. For example:

confluent-hub install confluentinc/kafka-connect-azure-event-hubs:1.1.1-preview

Adding a new connector plugin requires restarting Connect. Use the Confluent CLI command to restart Connect.

Tip

The command syntax for the Confluent CLI development commands changed in 5.3.0. These commands have been moved to confluent local. For example, the syntax for confluent start is now confluent local start. For more information, see confluent local.

confluent local stop connect && confluent local start connect

Your output should resemble:

Using CONFLUENT_CURRENT: /Users/username/Sandbox/confluent-snapshots/var/confluent.NuZHxXfq
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]

Check if the Azure Event Hubs plugin has been installed correctly and picked up by the plugin loader:

curl -sS localhost:8083/connector-plugins | jq .[].class | grep eventhubs
"io.confluent.connect.azure.eventhubs.EventHubsSourceConnector"

Azure Event Hubs Setup¶

Follow the official tutorial, create a resource group, then a namespace, then a event hub, and then produce events to the hub. The producer will need SAS info to get access to event hub.

Source Connector Configuration¶

Start the services using the Confluent CLI commands:

confluent local start

Create a configuration file named event-hubs-source-config.json with the following contents.

{
  "name": "EventHubsSourceConnector1",
  "config": {
    "confluent.topic.bootstrap.servers": "localhost:9092",
    "connector.class": "io.confluent.connect.azure.eventhubs.EventHubsSourceConnector",
    "kafka.topic": "event_hub_topic",
    "tasks.max": "1",
    "max.events": "1",
    "azure.eventhubs.sas.keyname": "<-Your Shared Access Policy name->",
    "azure.eventhubs.sas.key": "<-Your Shared Access key->",
    "azure.eventhubs.namespace": "<-Your namespace->",
    "azure.eventhubs.hub.name": "<-Your event hub->"
  }
}

The important configuration parameters used here are:

azure.eventhubs.hub.name: The event hub to subscribe to.
azure.eventhubs.namespace: The Event Hubs namespace where the source hub resides.
kafka.topic: The Kafka topic into which the events received from Event Hubs are produced.
tasks.max: The maximum number of tasks that should be created for this connector. Each task can be assigned with one or more event hub partition. The connector uses round-robin to assign partitions over tasks.
You must pass your shared access policy credentials to the Event Hubs connector through your source connector configuration. To pass SAS in the source configuration set the azure.eventhubs.sas.keyname and the azure.eventhubs.sas.key: parameters. You can look up the shared access key through Azure portal after a namespace has been created. Navigate to the namespace panel, click Shared access policies under Settings, and you can view a list of policies on the right. Click any one of the policies, and you will see both primary key and secondary key to the right.
```
"azure.eventhubs.sas.keyname":<-Your Shared Access Policy name->
"azure.eventhubs.sas.key":<-Your Shared Access key->
```

Run the following command to start the Azure Event Hubs source connector:

Caution

You must include a double dash (--) between the topic name and your flag. For more information, see this post.

confluent local load EventHubsSourceConnector1 -- -d event-hubs-source-config.json

To check that the connector started successfully view the Connect worker’s log by running:

confluent local log connect

Start a Kafka Consumer in a separate terminal session to view the data exported by the connector into the Kafka topic

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic event_hub_topic --from-beginning

Finally, stop the Confluent services using:

confluent local stop

Remove Unused Resources¶

To avoid any unintended charges, delete your resource group using the Azure portal. All the namespaces and event hubs in the resource group will be deleted as well.