.. _controlcenter_userguide_connect: Manage |kconnect-long| Using |c3-short| for |cp| ================================================ .. meta:: :title: Manage Kafka Connect cluster and connectors in Confluent Control Center :description: This document describes managing Kafka Connect clusters and connectors in Control Center. :keywords: Confluent Control Center, Connect, connectors, upload connector config file, download connector config file, connector status, pause connector, resume paused connector, edit connector, delete connector, RBAC |c3-short| provides the convenience of managing connectors for multiple |kconnect-long| clusters. Use |c3-short| to: - :ref:`Add a connector ` by completing UI fields. - Add a connector by :ref:`uploading a connector configuration file `. - :ref:`Download connector configuration files ` to reuse in another connector or cluster, or to use as a template. - :ref:`Edit a connector configuration ` and relaunch it. - :ref:`Pause a running connector `; resume a paused connector. - :ref:`Delete a connector `. - View the status of connectors in |kconnect| clusters. |c3-short| uses the |kconnect| API to get information on running connectors. For more information about |kconnect|, see :ref:`kafka_connect`. For more information about connectors, see :ref:`connectors-self-managed-cp`. .. tip:: If you are getting started learning about connectors, try out the :ref:`Kafka Connect Filestream Connectors` in a non-production (local) environment. Follow the instructions in :ref:`connect_quickstart`. .. _c3_config_connect_cluster: Configure |c3-short| for |kconnect| clusters -------------------------------------------- When you start a local |cp| install using the :confluent-cli:`confluent local services start|command-reference/local/services/confluent_local_services_start.html` command from the |confluent-cli|, a default |kconnect| cluster named **connect-default** is created for you. To configure |c3-short| to communicate with your own |kconnect| clusters in a production environment, set the ``confluent.controlcenter.connect..cluster=:`` option in the control center configuration file (``control-center.properties``). This option is required for connecting |c3-short| to your |kconnect| clusters. For the full configuration property definition, see the |kconnect| |c3| :ref:`configuration properties ` in the :ref:`controlcenter_configuration`. Multiple |kconnect-long| clusters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can configure |c3| to manage multiple |kconnect-long| clusters. For example, if you have a development |kconnect| cluster and a production |kconnect| cluster, you could add two ``confluent.controlcenter.connect..cluster=:`` references to the |c3-short| configuration properties file (``control-center.properties``). An example of this is shown below: .. code-block:: text ############################# Control Center Settings ############################# ... # A comma separated list of Connect host names confluent.controlcenter.connect.development.cluster=http://connect-dev:8083 confluent.controlcenter.connect.production.cluster=https://connect-prod:8443 ... For the full configuration property definition, see the |kconnect| |c3| :ref:`configuration properties ` in the :ref:`controlcenter_configuration`. .. _c3_all_connect_clusters_page: All |kconnect| Clusters page ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The **All Kafka Connect Clusters** page provides an overview of all |kconnect| clusters. To access the **All Kafka Connect Clusters** page: - Click the **Connect** menu for a cluster. - Click the **Connect** panel on the cluster **Overview** page. Use the **All Kafka Connect Clusters** page to: - Search for a |kconnect| cluster by its cluster name and ID. - Sort a column by clicking in the column title cell. - Navigate to the :ref:`c3_connectors_page`. - View a breakdown for each |kconnect| cluster by connector status. The page shows the total number of running, degraded, failed, and paused connectors. .. figure:: ../images/c3-all-connect-clusters-page.png :width: 600px .. _c3_connectors_page: Connectors page ^^^^^^^^^^^^^^^ The **Connectors** page provides an overview of the connectors configured for a |kconnect| cluster. To access the **Connectors** page, click the **Cluster name** link in the :Ref:`c3_all_connect_clusters_page` for a |kconnect| cluster. Use the **Connectors** page to: - View a summary of connectors for the |kconnect| cluster. - Search for a connector by its name and ID. - Filter the view of connectors by Source or Sink type. - :ref:`c3-add-connector` by either completing the UI fields manually or :ref:`uploading a connector configuration file `. - Sort a column by clicking in the column title cell. - :ref:`c3-add-connector` to the |kconnect| cluster. - Navigate to view connector details, :ref:`edit a connector `, or :ref:`delete a connector `. .. figure:: ../images/c3-connectors-page.png :width: 600px The **Status**, **Name**, **Category**, **Type**, **Topics** (for sink connectors only), and **Number of Tasks** are shown for each configured connector. .. note:: When reducing the number of tasks in a connector's configuration (for example, from 32 to 30 or from 36 to 12) the **Number of Tasks** field will display similar to the following: .. figure:: ../images/c3-connector-number-of-tasks.png :width: 100px :height: 100px .. _c3-add-connector: Add a connector (non-|rbac| environment) ---------------------------------------- Follow these steps to configure a source or sink connector by completing the applicable UI fields. For details about connector settings common and unique to source and sink connectors, see :connect-common:`Configuring Connectors|configuring.html`. This procedure is applicable to a non-|rbac| workflow. For an |rbac| workflow, see :ref:`c3-add-connector-rbac-workflow`). .. important:: In the |cp| 6.0 release, some connectors that were previously bundled with |cp| are now available through the `Confluent Hub `__. For details, see the page on :connect-common:`Supported Connectors|supported.html`. There are two steps (tabs) to complete in this workflow: #. Set up the connection. #. Test and verify. .. tip:: You can also add a connector by :ref:`uploading a connector configuration file `. #. Select a cluster from the navigation bar and click the **Connect** menu. The :ref:`c3_all_connect_clusters_page` opens. .. figure:: ../images/c3-all-connect-clusters-page.png :width: 600px #. In the **Cluster name** column, click the **connect-default** link (or the link for your |kconnect| cluster). The :ref:`c3_connectors_page` opens. .. figure:: ../images/c3-no-connectors.png :width: 600px #. Click **Add connector**. The Browse page for selecting connectors opens. The connectors that initially appear in this page are :connect-common:`bundled|supported.html` with |cp|. .. figure:: ../images/c3-connect-select-connector.png :width: 700px .. tip:: From the **Filter by type** menu, select either **Sources** or **Sinks** to narrow the available selections for the connector type. .. figure:: ../images/c3-filter-source-sink.png :scale: 75% #. Click the tile for the connector you want to configure. The **Add Connector** page opens to the **01 Setup Connection** tab. .. figure:: ../images/c3-add-connector.png :width: 700px .. tip:: There are many available fields for configuring connectors. Click a link in the shortcut panel to quickly jump to a section. The sections shown depend on the connector you are configuring. #. Complete the fields as appropriate for the connector. Required fields are indicated with an asterisk. .. note:: When configuring the `Generalized Google Cloud Storage Source connector `__, you won't be able to add the ``topic.regex.list`` configuration parameter if the mode for the connector is set to ``RESTORE_BACKUP``, which is the default mode. If you set the mode to ``GENERIC``, you will see ``topic.regex.list`` listed as an option under **Kafka Topic Regex** in the **Topic** section. For more details about each of these parameters, see the `Generalized GCS Source Connector Configuration Properties `__ page. .. _c3-add-connector-prop-step: #. (Optional) If there are additional properties you need to add, click **Add a property**. The **Additional Properties** dialog opens for you to enter the property name. After entering the property name, enter the value for the property. .. figure:: ../images/c3-connector-add-property.png :scale: 50% Add a property link .. figure:: ../images/c3-connector-add-prop-modal.png :scale: 50% Add property dialog .. figure:: ../images/c3-connector-addl-props.png :scale: 50% Added property .. tip:: Click the trash icon to delete an additional property. You have the option to undo the operation. #. Click **Continue**. The **02 Test and verify** page opens (non-|rbac| worfklow). .. note:: For an |rbac| workflow, the 02 Security page opens. For more information, see :ref:`c3-add-connector-rbac-workflow`. .. figure:: ../images/c3-connect-download-config-link.png :width: 600px #. (Optional) Click **Download connector config file**. See :ref:`c3_download_connector_config` for details. #. Review the information and click **Launch**. The information displayed is sent to the :ref:`Connect REST API `. - If the configuration was successful, the connector appears in the connectors table within the :ref:`c3_connectors_page`. Green bars indicate the connector is running. - If the configuration was unsuccessful, the **Status** column indicates Failed. Red bars indicate the connector is not running. In the **Name** column, click the link for the connector and edit the configuration fields. Repeat the process as necessary. .. _c3-add-connector-rbac-workflow: Add a connector (|rbac| environment) ------------------------------------ Follow this procedure to add a connector in |c3-short| when |rbac| is enabled. .. important:: In the |cp| 6.0 release, some connectors that were previously bundled with |cp| are now available through the `Confluent Hub `__. For details, see the page on :connect-common:`Supported Connectors|supported.html`. There are typically three steps (tabs) to adding a connector when |rbac| is enabled: #. Set up the connection #. Configure security (|rbac|) #. Test and verify To use this procedure, your connector configuration must use the standard OAuthBearer SASL mechanism, and your worker configuration must not contain principal credentials. Otherwise, skip this procedure (02 Security page). .. important:: If your :ref:`worker configuration ` does not use the OAuthBearer SASL mechanism, do not enter the Connector Username and Password fields in the 02 Security tab; the connector configuration will not work if you do. If you configure a connector to use a different authentication mechanism, it should work as long as that mechanism is enabled on the |ak| cluster. .. tip:: If you use the OAuthBearer SASL mechanism, it's recommended that you exclude the ``producer.sasl.jaas.config`` and ``consumer.sasl.jaas.config`` properties from your worker file. Instead, specify those properties on a per-connector basis with the ``producer.override.sasl.jaas.config`` and ``consumer.override.sasl.jaas.config`` properties. To specify these configurations in a connector, use the **Additional Properties** fields in the **Add Connector > 01 Setup Connection** tab. For instructions on adding a property, see the :ref:`additional properties step `. If you specify these additional properties, do not enter the Connector Username and Password fields in the Add Connector page 02 Security tab. Simply click **Continue** to skip the 02 Security tab and go directly to the 03 Test and Verify tab. **To add a connector in a standard RBAC environment** Complete the steps for adding a connector as described in the non-|rbac| workflow, :ref:`c3-add-connector`. This procedure describes the details for the **02 Security** step in the |rbac| connector workflow. .. login include .. include:: security/includes/c3-login.rst #. Follow steps 1 through 7 in :ref:`c3-add-connector`. #. (Optional/Conditional) In the step 02 Security page, enter your credentials: - Connector Username (``principal.service.name``) - Connector Password (``principal.service.password``) .. figure:: ../images/c3-rbac-connector-step3-collapsed.png :width: 600px Add connector RBAC security workflow collapsed page view The ``principal.service.name`` and ``principal.service.password`` are converted into ``producer.override.sasl.jaas.config`` and ``consumer.override.sasl.jaas.config``. #. (Optional) If you do not have the information, click the **How do I get these credentials?** link. The page expands. .. figure:: ../images/c3-rbac-connector-template-upper.png :width: 600px Add Connector RBAC Security Workflow expanded upper page .. figure:: ../images/c3-rbac-connector-template-lower.png :width: 600px Add connector RBAC security workflow expanded lower page view a. Review the information presented on the page, including the documentation links. b. Click **Copy to Clipboard**. Send the information to your RBAC administrator so that they can grant you the necessary privileges. .. tip:: Your |rbac| administrator can also create role bindings using the |c3-short| UI rather than the CLI. See :ref:`c3_rbac_manage_roles_ui`. c. When you receive the information from your RBAC administrator, complete your credentials. #. Click **Continue**. The 03 Test and verify page opens. #. Review the information and click **Launch**. .. _c3_download_connector_config: Download a connector configuration file --------------------------------------- Download a connector configuration file while you are :ref:`adding a connector `. After downloading the configuration file, you can upload the configuration file into another |kconnect| cluster or use it as a template for another connector in the same or another |kconnect| cluster. Supported formats for download: - ``.json`` file - ``.properties`` file #. Select a cluster from the navigation bar and click the **Connect** menu. The :ref:`c3_all_connect_clusters_page` opens. #. In the **Cluster name** column, click the **connect-default** link (or the link for your |kconnect| cluster). The :ref:`c3_connectors_page` opens. .. tip: If a connector has already been created, navigate to its **Settings** page to access the download link. #. Click **Add connector**. Complete the fields as described in :ref:`adding a connector `. Continue to the Test and verify page. .. figure:: ../images/c3-connect-download-config-link.png :width: 600px #. Click **Download connector config file**. The Download config file dialog opens. .. figure:: ../images/c3-connector-download-config.png :scale: 50% #. Click the file type to download: **Download .JSON** or **Download .properties**. The file is downloaded into your downloads directory. When you are done downloading files, click **x** to close the download dialog. Example JSON connector config file: .. code-block:: JSON { "name": "datagen-users", "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "kafka.topic": "users", "max.interval": "990", "iterations": "1000000000", "quickstart": "users" } Example properties connector config file: .. sourcecode:: bash name = datagen-users connector.class = io.confluent.kafka.connect.datagen.DatagenConnector key.converter = org.apache.kafka.connect.storage.StringConverter kafka.topic = users max.interval = 990 iterations = 1000000000 quickstart = users #. Click **Launch** to launch the connector. .. logged MMA-5482 to add a close button .. _c3_upload_connector_config: Upload a connector configuration file to add a connector -------------------------------------------------------- Upload a connector configuration file to add a connector. You can upload a connector config file from either the Browse connectors page when adding a connector, or directly from the :ref:`c3_connectors_page`. .. .. figure:: ../images/c3-connect-upload-config-browse.png .. :width: 600px Upload config from Browse connectors Supported formats for upload: - ``.json`` file - ``.properties`` file #. Select a cluster from the navigation bar and click the **Connect** menu. The :ref:`c3_all_connect_clusters_page` opens. #. In the **Cluster name** column, click the **connect-default** link (or the link for your |kconnect| cluster). The :ref:`c3_connectors_page` opens. .. figure:: ../images/c3-connect-upload-config-connectors.png :width: 700px Upload config from Connectors page #. Click **+ Upload connector config file**. #. In the **Downloads** directory (or navigate to the directory where your config resides), select the configuration file you want to upload and click **Open**. - The Connector UI fields are populated with the entries in the configuration file. - If an error is displayed, make sure the file has valid JSON if applicable. Ensure the JSON or properties file has valid properties for the type of connector. .. _c3_edit_connector: Edit a connector ---------------- :ref:`Pause ` the connector before making configuration changes. #. Select a cluster from the navigation bar and click the **Connect** menu. The :ref:`c3_all_connect_clusters_page` opens. #. In the **Cluster name** column, click the **connect-default** link (or the link for your |kconnect| cluster). The :ref:`c3_connectors_page` opens. #. Click the **Name** column for the connector. #. Click the **Settings** tab. .. tip:: Click the **Overview** tab to view a summary. #. Make the changes you want to the connector fields. .. tip:: Hover on a field info (i) icon to get more information about the field. .. note:: The Connector class and name fields are locked and cannot be changed. #. Click **Continue**. #. (Optional) Click **Download connector config file**. See :ref:`c3_download_connector_config` for details. #. Review the information and click **Launch**. .. _c3_pause_resume_connector: Pause and resume a running connector ------------------------------------ Pause a running connector if you need to troubleshoot or make configuration changes. You can pause or resume a connector from either the **Overview** or the **Settings** pages. The **Overview** and **Connectors** pages display ``Paused`` in the **Connectors** summary panel and the **Status** column for a paused connector. See the following example from the **Connectors** page: .. figure:: /images/c3-paused-status-connectors-page.png :width: 600px #. :ref:`c3_edit_connector`. The **Overview** page opens. #. Click the **Pause** icon to pause a running connector. #. Click the resume (play) icon to allow the connector to run again. .. tip:: You can also :ref:`delete a connector ` when :ref:`editing a connector ` by clicking the trash can icon. You are prompted to confirm deleting the connector. .. _c3_restart_connector: Restart a connector ------------------- Restart a connector if you need to reboot the connector and all its tasks. You can restart the connector by clicking **Restart** on the **Overview** page. .. figure:: /images/c3-connect-restart.png :width: 150px :height: 150px #. :ref:`c3_edit_connector`. The **Overview** page opens. #. Click **Restart** to restart the connector. #. Verify the tasks have restarted and transition to the **Runnning** state. Restart failed components ^^^^^^^^^^^^^^^^^^^^^^^^^ In the event that some or all of a connector's tasks have failed, and/or the connector has failed, |c3-short| will display a message in the **Overview** page. If there is a failure, |c3-short| will display a message notifying one of the following: - Tasks have failed. To restart the failed tasks, click **Restart Failed Tasks** to reboot only the failed tasks as shown in the following example: .. figure:: /images/c3-connect-failed-tasks.png :width: 450px :height: 450px - The connector has failed. To restart only the failed connector, click **Restart Failed Connector**. - The connector and tasks have failed. To restart the failed connector and tasks, click **Restart Failed Connector And Tasks**. Be sure to verify the failed components have restarted and transitioned to the **Running** state. .. _c3_delete_connector: Delete a connector ------------------ Delete a connector that is no longer needed. You can :ref:`pause ` a connector before deleting it. Deleting a connector stops a connector if it is running. You can delete a connector from the **Overview** or **Settings** pages. #. Select a cluster from the navigation bar and click the **Connect** menu. The :ref:`c3_all_connect_clusters_page` opens. #. In the **Cluster name** column, click the **connect-default** link. The :ref:`c3_connectors_page` opens. #. Click the **Name** column for the connector. The **Overview** page opens. #. Click the trash icon. You are prompted to confirm the deletion. #. Click **Confirm**.