.. _event-hubs-source-connector: |az| Event Hubs Source Connector for |cp| ========================================= The |kconnect-long| |az| Event Hubs Source Connector is used to poll data from |az| Event Hubs and persist the data to an |ak-tm| topic. Install the |az| Event Hubs Connector ------------------------------------- .. include:: ../includes/connector-install.rst .. include:: ../includes/connector-install-hub.rst .. codewithvars:: bash confluent-hub install confluentinc/kafka-connect-azure-event-hubs:latest .. include:: ../includes/connector-install-version.rst .. codewithvars:: bash confluent-hub install confluentinc/kafka-connect-azure-event-hubs:1.1.4 -------------------------- Install Connector Manually -------------------------- `Download and extract the ZIP file `_ for your connector and then follow the manual connector installation :ref:`instructions `. License ------- .. include:: ../includes/enterprise-license.rst See :ref:`azure-event-hubs-source-connector-license-config` for license properties and :ref:`event-hubs-source-license-topic-configuration` for information about the license topic. Usage Notes ----------- |az| Event Hubs use ``Shared Access Signature (SAS)`` for authentication and authorization. For more information, see the `Event Hubs Authentication and Security Model Overview `_. ------------------ REST-based example ------------------ This configuration is used typically along with :ref:`distributed workers `. Write the following JSON to ``connector.json``, configure all of the required values, and use the command below to post the configuration to one the distributed connect workers. Refer to :ref:`REST API ` for more information about the |kconnect-long|. **Connect Distributed REST example:** .. codewithvars:: json :emphasize-lines: 8,9 { "name": "EventHubsSourceConnector1", "config": { "confluent.topic.bootstrap.servers": "< Required Configuration >", "connector.class": "io.confluent.connect.azure.eventhubs.EventHubsSourceConnector", "kafka.topic": "< Required Configuration >", "tasks.max": "1", "max.events": "< Optional Configuration >", "azure.eventhubs.sas.keyname": "< Required Configuration >", "azure.eventhubs.sas.key": "< Required Configuration >", "azure.eventhubs.namespace": "< Required Configuration >", "azure.eventhubs.hub.name": "< Required Configuration >" } } Use curl to post the configuration to one of the |kconnect-long| Workers. Change `http://localhost:8083/` to the endpoint of one of your |kconnect-long| workers. **Create a new connector:** .. codewithvars:: bash curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors **Update an existing connector:** .. codewithvars:: bash curl -s -X PUT -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors/EventHubsSourceConnector1/config .. _event_hubs_quickstart: Quick Start ----------- The |az| Event Hubs source connector is used to poll data from an Event Hub, and write into a |ak| topic. Before you begin, you need to have an |az| subscription with the privilege to create resource group and service. Using the |az| portal, create a namespace and event hub. Then produce some events to the hub using Event Hubs API. ----------------- Preliminary Setup ----------------- .. include:: ../../includes/install-cli-prereqs.rst Navigate to your Confluent Platform installation directory and run the following command to install the latest connector version: :: confluent-hub install confluentinc/kafka-connect-azure-event-hubs:latest You can install a specific version by replacing latest with a version number. For example: :: confluent-hub install confluentinc/kafka-connect-azure-event-hubs:1.1.1-preview Adding a new connector plugin requires restarting Connect. Use the :ref:`Confluent CLI ` command to restart |kconnect|. .. include:: ../../includes/cli-new.rst .. codewithvars:: bash $ |confluent_stop| connect && |confluent_start| connect Using CONFLUENT_CURRENT: /Users/username/Sandbox/confluent-snapshots/var/confluent.NuZHxXfq Starting zookeeper zookeeper is [UP] Starting kafka kafka is [UP] Starting schema-registry schema-registry is [UP] Starting kafka-rest kafka-rest is [UP] Starting connect connect is [UP] Check if the |az| Event Hubs plugin has been installed correctly and picked up by the plugin loader: :: $ curl -sS localhost:8083/connector-plugins | jq .[].class | grep eventhubs "io.confluent.connect.azure.eventhubs.EventHubsSourceConnector" --------------------- |az| Event Hubs Setup --------------------- Follow the `official tutorial`_, create a `resource group`_, then a `namespace`_, then a `event hub`_, and then `produce events`_ to the hub. The producer will need `SAS`_ info to get access to event hub. ------------------------------ Source Connector Configuration ------------------------------ Start the services using the :ref:`Confluent CLI ` commands: .. codewithvars:: bash |confluent_start| Create a configuration file named event-hubs-source-config.json with the following contents. :: { "name": "EventHubsSourceConnector1", "config": { "confluent.topic.bootstrap.servers": "localhost:9092", "connector.class": "io.confluent.connect.azure.eventhubs.EventHubsSourceConnector", "kafka.topic": "event_hub_topic", "tasks.max": "1", "max.events": "1", "azure.eventhubs.sas.keyname": "RootManageSharedAccessKey", "azure.eventhubs.sas.key": "<-Your primary SAS key->", "azure.eventhubs.namespace": "<-Your namespace->", "azure.eventhubs.hub.name": "<-Your event hub->" } } The important configuration parameters used here are: * ``azure.eventhubs.hub.name``: The event hub to subscribe to. * ``azure.eventhubs.namespace``: The Event Hubs namespace where the source hub resides. * ``kafka.topic``: The |ak| topic into which the events received from Event Hubs are produced. * ``tasks.max``: The maximum number of tasks that should be created for this connector. Each task can be assigned with one or more `event hub partition`_. The connector uses round-robin to assign partitions over tasks. * You must pass your SAS credentials to the Event Hubs connector through your source connector configuration. To pass SAS in the source configuration set the `azure.eventhubs.sas.keyname` and the `azure.eventhubs.sas.key`: parameters. You can look up the primary SAS key through |az| portal after a namespace has been created. Navigate to the namespace panel, click *Shared access policies* under **Settings**, and you can view a list of policies on the right. Click any one of the policies, and you will see both primary key and secondary key to the right. :: "azure.eventhubs.sas.keyname":<-Your Shared Access Policy name-> "azure.eventhubs.sas.key":<-Your primary SAS key-> Run the following command to start the |az| Event Hubs source connector: .. include:: ../../includes/confluent-local-consume-limit.rst .. codewithvars:: bash |confluent_load| EventHubsSourceConnector1|dash| -d event-hubs-source-config.json To check that the connector started successfully view the Connect worker's log by running: .. codewithvars:: bash |confluent_log| connect Start a Kafka Consumer in a separate terminal session to view the data exported by the connector into the |ak| topic :: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic event_hub_topic --from-beginning Finally, stop the Confluent services using: .. codewithvars:: bash |confluent_stop| ----------------------- Remove Unused Resources ----------------------- To avoid any unintended charges, delete your resource group using the |az| portal. All the namespaces and event hubs in the resource group will be deleted as well. .. _official tutorial: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create .. _produce events: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-send .. _resource group: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview .. _namespace: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features .. _event hub: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features .. _event hub partition: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features .. _SAS: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-authentication-and-security-model-overview Additional Documentation ------------------------ .. toctree:: :maxdepth: 1 azure_event_hubs_source_connector_config changelog