.. _event-hubs-source-connector:

|az| Event Hubs Source Connector for |cp|
=========================================

The |kconnect-long| |az| Event Hubs Source Connector is used to poll data from |az| Event Hubs and persist the data to an |ak-tm| topic.

Install the |az| Event Hubs Connector
-------------------------------------

.. include:: ../includes/connector-install.rst

.. include:: ../includes/connector-install-hub.rst

.. codewithvars:: bash

   confluent-hub install confluentinc/kafka-connect-azure-event-hubs:latest

.. include:: ../includes/connector-install-version.rst

.. codewithvars:: bash

   confluent-hub install confluentinc/kafka-connect-azure-event-hubs:1.1.4

--------------------------
Install Connector Manually
--------------------------

`Download and extract the ZIP file <https://www.confluent.io/connector/kafka-connect-azure-event-hubs/#download>`_ for your
connector and then follow the manual connector installation :ref:`instructions <connect_install_connectors>`.

License
-------

.. include:: ../includes/enterprise-license.rst

See :ref:`azure-event-hubs-source-connector-license-config` for license properties and :ref:`event-hubs-source-license-topic-configuration` for information about the license topic.


Usage Notes
-----------

|az| Event Hubs use ``Shared Access Signature (SAS)`` for authentication and authorization. For more information, see the
`Event Hubs Authentication and Security Model Overview <https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-authentication-and-security-model-overview>`_.

------------------
REST-based example
------------------


This configuration is used typically along with :ref:`distributed workers <distributed-workers>`.
Write the following JSON to ``connector.json``, configure all of the required values, and use the command below to
post the configuration to one the distributed connect workers. Refer to :ref:`REST API <connect_userguide_rest>` for more information about the |kconnect-long|.

**Connect Distributed REST example:**

.. codewithvars:: json
   :emphasize-lines: 8,9

    {
      "name": "EventHubsSourceConnector1",
      "config": {
        "confluent.topic.bootstrap.servers": "< Required Configuration >",
        "connector.class": "io.confluent.connect.azure.eventhubs.EventHubsSourceConnector",
        "kafka.topic": "< Required Configuration >",
        "tasks.max": "1",
        "max.events": "< Optional Configuration >",
        "azure.eventhubs.sas.keyname": "< Required Configuration >",
        "azure.eventhubs.sas.key": "< Required Configuration >",
        "azure.eventhubs.namespace": "< Required Configuration >",
        "azure.eventhubs.hub.name": "< Required Configuration >"
      }
    }

Use curl to post the configuration to one of the |kconnect-long| Workers. Change `http://localhost:8083/` to the endpoint of
one of your |kconnect-long| workers.

**Create a new connector:**

.. codewithvars:: bash

    curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors

**Update an existing connector:**

.. codewithvars:: bash

    curl -s -X PUT -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors/EventHubsSourceConnector1/config
    
.. _event_hubs_quickstart:

Quick Start
-----------

The |az| Event Hubs source connector is used to poll data from an Event Hub, and write into a |ak| topic. Before
you begin, you need to have an |az| subscription with the privilege to create resource group and service.
Using the |az| portal, create a namespace and event hub. Then produce some events to the hub using Event Hubs API.

-----------------
Preliminary Setup
-----------------

.. include:: ../../includes/install-cli-prereqs.rst

Navigate to your Confluent Platform installation directory and run the following
command to install the latest connector version:

::

   confluent-hub install confluentinc/kafka-connect-azure-event-hubs:latest

You can install a specific version by replacing latest with a version
number. For example:

::

   confluent-hub install confluentinc/kafka-connect-azure-event-hubs:1.1.1-preview

Adding a new connector plugin requires restarting Connect. Use the
:ref:`Confluent CLI <cli>` command to restart |kconnect|.

.. include:: ../../includes/cli-new.rst

.. codewithvars:: bash

    $ |confluent_stop| connect && |confluent_start| connect
    Using CONFLUENT_CURRENT: /Users/username/Sandbox/confluent-snapshots/var/confluent.NuZHxXfq
    Starting zookeeper
    zookeeper is [UP]
    Starting kafka
    kafka is [UP]
    Starting schema-registry
    schema-registry is [UP]
    Starting kafka-rest
    kafka-rest is [UP]
    Starting connect
    connect is [UP]

Check if the |az| Event Hubs plugin has been installed correctly and picked up
by the plugin loader:

::

    $ curl -sS localhost:8083/connector-plugins | jq .[].class | grep eventhubs
    "io.confluent.connect.azure.eventhubs.EventHubsSourceConnector"

---------------------
|az| Event Hubs Setup
---------------------

Follow the `official tutorial`_, create a `resource group`_, then a `namespace`_,
then a `event hub`_, and then `produce events`_ to the hub. The producer will need `SAS`_ info to
get access to event hub.

------------------------------
Source Connector Configuration
------------------------------

Start the services using the :ref:`Confluent CLI <cli>` commands:

.. codewithvars:: bash

 |confluent_start|

Create a configuration file named event-hubs-source-config.json with the following contents.


::

  {
    "name": "EventHubsSourceConnector1",
    "config": {
      "confluent.topic.bootstrap.servers": "localhost:9092",
      "connector.class": "io.confluent.connect.azure.eventhubs.EventHubsSourceConnector",
      "kafka.topic": "event_hub_topic",
      "tasks.max": "1",
      "max.events": "1",
      "azure.eventhubs.sas.keyname": "RootManageSharedAccessKey",
      "azure.eventhubs.sas.key": "<-Your primary SAS key->",
      "azure.eventhubs.namespace": "<-Your namespace->",
      "azure.eventhubs.hub.name": "<-Your event hub->"
    }
  }

The important configuration parameters used here are:

* ``azure.eventhubs.hub.name``: The event hub to subscribe to.
* ``azure.eventhubs.namespace``: The Event Hubs namespace where the source hub resides.
* ``kafka.topic``: The |ak| topic into which the events received from Event Hubs are produced.
* ``tasks.max``: The maximum number of tasks that should be created for this connector. Each task can be assigned with one or more `event hub partition`_. The connector uses round-robin to assign partitions over tasks.
* You must pass your SAS credentials to the Event Hubs connector through your source connector configuration. To pass SAS in the source configuration set the `azure.eventhubs.sas.keyname` and the `azure.eventhubs.sas.key`: parameters. You can look up the primary SAS key through |az| portal after a namespace has been created. Navigate to the namespace panel, click *Shared access policies* under **Settings**, and you can view a list of policies on the right. Click any one of the policies, and you will see both primary key and secondary key to the right.

  ::
     
     "azure.eventhubs.sas.keyname":<-Your Shared Access Policy name->
     "azure.eventhubs.sas.key":<-Your primary SAS key->

Run the following command to start the |az| Event Hubs source connector:

.. include:: ../../includes/confluent-local-consume-limit.rst

.. codewithvars:: bash

 |confluent_load| EventHubsSourceConnector1|dash| -d event-hubs-source-config.json

To check that the connector started successfully view the Connect
worker's log by running:

.. codewithvars:: bash

 |confluent_log| connect

Start a Kafka Consumer in a separate terminal session to view the data exported by
the connector into the |ak| topic

::

  bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic event_hub_topic --from-beginning

Finally, stop the Confluent services using:

.. codewithvars:: bash

 |confluent_stop|

-----------------------
Remove Unused Resources
-----------------------

To avoid any unintended charges, delete your resource group using the |az| portal. All the namespaces and event hubs in the resource group will be deleted as well.

.. _official tutorial: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create
.. _produce events: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-send
.. _resource group: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview
.. _namespace: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features
.. _event hub: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features
.. _event hub partition: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features
.. _SAS: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-authentication-and-security-model-overview

Additional Documentation
------------------------

.. toctree::
   :maxdepth: 1

   azure_event_hubs_source_connector_config
   changelog