Manage Kafka Connect Using Confluent Control Center for Confluent Platform

Confluent Control Center provides the convenience of managing connectors for multiple Kafka Connect clusters. Use Control Center to do any of the following:

• Add a connector by completing UI fields.
• Add a connector by uploading a connector configuration file.
• Download connector configuration files to reuse in another connector or cluster, or to use as a template.
• Edit a connector configuration and relaunch it.
• Pause a running connector; resume a paused connector.
• Delete a connector.
• View the status of connectors in Connect clusters.

Control Center uses the Connect API to get information on running connectors.

For more information about Connect, see Kafka Connect. For more information about connectors, see Kafka Connectors.

Tip

If you are getting started learning about connectors, try out the Kafka Connect Filestream Connectors in a non-production (local) environment. Follow the instructions in Quick Start: Move Data In and Out of Kafka with Kafka Connect.

Configure Control Center for Connect clusters

When you start a local Confluent Platform install using the confluent local services start command from the Confluent CLI, a default Connect cluster named connect-default is created for you. To configure Control Center to communicate with your own Connect clusters in a production environment, set the confluent.controlcenter.connect.<connect-cluster-name>.cluster=<host>:<port> option in the control center configuration file (control-center.properties). This option is required for connecting Control Center to your Connect clusters. For the full configuration property definition, see the Connect Confluent Control Center configuration properties in the Control Center Configuration Reference for Confluent Platform.

Multiple Kafka Connect clusters

You can configure Confluent Control Center to manage multiple Kafka Connect clusters. For example, if you have a development Connect cluster and a production Connect cluster, you could add two confluent.controlcenter.connect.<connect-cluster-name>.cluster=<host>:<port> references to the Control Center configuration properties file (control-center.properties). An example of this is shown below:

############################# Control Center Settings #############################

...

# A comma separated list of Connect host names
confluent.controlcenter.connect.development.cluster=http://connect-dev:8083
confluent.controlcenter.connect.production.cluster=https://connect-prod:8443

...

For the full configuration property definition, see the Connect Confluent Control Center configuration properties in the Control Center Configuration Reference for Confluent Platform.

All Connect Clusters page

The All Kafka Connect Clusters page provides an overview of all Connect clusters.

To access the All Kafka Connect Clusters page:

• Click the Connect menu for a cluster.
• Click the Connect panel on the cluster Overview page.

Use the All Kafka Connect Clusters page to:

• Search for a Connect cluster by its cluster name and ID.
• Sort a column by clicking in the column title cell.
• Navigate to the Connectors page.
• View a breakdown for each Connect cluster by connector status.

The page shows the total number of running, degraded, failed, and paused connectors.

Connectors page

The Connectors page provides an overview of the connectors configured for a Connect cluster.

To access the Connectors page, click the Cluster name link in the All Connect Clusters page for a Connect cluster.

Use the Connectors page to:

• View a summary of connectors for the Connect cluster.
• Search for a connector by its name and ID.
• Filter the view of connectors by Source or Sink type.
• Add a connector (non-RBAC environment) by either completing the UI fields manually or uploading a connector configuration file.
• Sort a column by clicking in the column title cell.
• Add a connector (non-RBAC environment) to the Connect cluster.
• Navigate to view connector details, edit a connector, or delete a connector.

The Status, Name, Category, Type, Topics (for sink connectors only), and Number of Tasks are shown for each configured connector.

When reducing the number of tasks in a connector’s configuration (for example, from 32 to 30 or from 36 to 12) the Number of Tasks field will display similar to the following:

Add a connector (non-RBAC environment)

Follow these steps to configure a source or sink connector by completing the applicable UI fields. You can also add a connector by uploading a connector configuration file. For details about connector settings common and unique to source and sink connectors, see Configuring Connectors.

This procedure is applicable to a non-RBAC workflow. For an role-based access control (RBAC) workflow, see Add a connector (RBAC environment)).

There are two steps (tabs) to complete in this workflow:

• Set up the connection.
• Test and verify.

To add a connector in a non-RBAC environment

1. Select a cluster from the navigation bar and click the Connect menu. The All Connect Clusters page opens.

   

2. In the Cluster name column, click the connect-default link (or the link for your Connect cluster). The Connectors page opens.

   

3. Click Add connector. The Browse page for selecting connectors opens. The connectors that initially appear in this page are bundled with Confluent Platform. To narrow the available selections, select either Sources or Sinks from the Filter by type menu.

   

4. Click the tile for the connector you want to configure. The Add Connector page opens to the 01 Setup Connection tab. Use the shortcut panel to the right to navigate the list of configurations.

   

5. Complete the fields as appropriate for the connector. Required fields are indicated with an asterisk.

   Generalized GCS Source connector configuration

   When configuring the Generalized Google Cloud Storage Source connector, you won’t be able to add the topic.regex.list configuration parameter if the mode for the connector is set to RESTORE_BACKUP, which is the default mode. If you set the mode to GENERIC, you will see topic.regex.list listed as an option under Kafka Topic Regex in the Topic section. For more details about each of these parameters, see the Generalized GCS Source Connector Configuration Properties page.

6. (Optional) If there are additional properties you need to add, click Add a property. The Additional Properties dialog opens for you to enter the property name. After entering the property name, enter the value for the property.

   

   Add a property link

   

   Add property dialog

   

   Added property

   To delete a property, click the trash icon. You can undo the operation.

7. Click Continue. The 02 Test and verify page opens. (If the 02 Security page opens, see RBAC workflow.)

   

8. (Optional) Click Download connector config file. See Download a connector configuration file for details.

9. Review the information and click Launch. The information displayed is sent to the Connect REST API.

   • If the configuration was successful, the connector appears in the connectors table within the Connectors page. Green bars indicate the connector is running.
   • If the configuration was unsuccessful, the Status column indicates Failed. Red bars indicate the connector is not running. In the Name column, click the link for the connector and edit the configuration fields. Repeat the process as necessary.

Add a connector (RBAC environment)

Follow this procedure to add a connector in Control Center when RBAC is enabled.

There are typically three steps (tabs) to adding a connector when RBAC is enabled:

1. Set up the connection
2. Configure security (RBAC)
3. Test and verify

To use this procedure, your connector configuration must use the standard SASL/OAUTHBEARER mechanism, and your worker configuration must not contain principal credentials. Otherwise, skip this procedure (02 Security page).

The SASL/OAUTHBEARER mechanism supports two types of user stores:

• LDAP based
• OAuth client credentials based

To use the SASL/OAUTHBEARER mechanism, exclude the producer.sasl.jaas.config and consumer.sasl.jaas.config properties from your worker file. Instead, specify those properties on a per-connector basis with the producer.override.sasl.jaas.config and consumer.override.sasl.jaas.config properties. To specify these configurations in a connector, use the Additional Properties fields in the Add Connector > 01 Setup Connection tab. For instructions on adding a property, see the additional properties step. If you specify these additional properties, do not enter the Connector Username and Password fields in the Add Connector page 02 Security tab. Simply click Continue to skip the 02 Security tab and go directly to the 03 Test and Verify tab.

Important

If your worker configuration does not use the SASL/OAUTHBEARER LDAP-based mechanism, do not enter the Connector Username and Password fields in the 02 Security tab; the connector configuration will not work if you do. If you configure a connector to use a different authentication mechanism, it should work as long as that mechanism is enabled on the Kafka cluster.

To add a connector in a standard RBAC environment

Complete the steps for adding a connector as described in the non-RBAC workflow, Add a connector (non-RBAC environment). This procedure describes the details for the 02 Security step in the RBAC connector workflow.

1. Log in to Control Center.

2. Follow steps 1 through 7 in Add a connector (non-RBAC environment).

3. (Optional/Conditional) Take one of the following actions, depending on the user store your connector uses:

   • For connectors using OAuth client credentials based configs, skip this step.

   • For connectors using LDAP-based credentials, in the step 02 Security page, enter your credentials:

     • Connector Username (principal.service.name)
     • Connector Password (principal.service.password)
     

     Add connector RBAC security workflow collapsed page view

     The principal.service.name and principal.service.password are converted into producer.override.sasl.jaas.config and consumer.override.sasl.jaas.config.

4. (Optional) If you do not have the information, click the How do I get these credentials? link. The page expands.

   

   Add Connector RBAC Security Workflow expanded upper page

   

   Add connector RBAC security workflow expanded lower page view

   1. Review the information presented on the page, including the documentation links.

   2. Click Copy to Clipboard. Send the information to your RBAC administrator so that they can grant you the necessary privileges.

      Tip

      Your RBAC administrator can also create role bindings using the Control Center UI rather than the CLI. See Manage RBAC roles with Control Center on Confluent Platform.

   3. When you receive the information from your RBAC administrator, complete your credentials.

5. Click Continue. The 03 Test and verify page opens.

6. Review the information and click Launch.

Download a connector configuration file

Download a connector configuration file while you are adding a connector. After downloading the configuration file, you can upload the configuration file into another Connect cluster or use it as a template for another connector in the same or another Connect cluster.

Supported formats for download:

• .json file
• .properties file

1. Select a cluster from the navigation bar and click the Connect menu. The All Connect Clusters page opens.

2. In the Cluster name column, click the connect-default link (or the link for your Connect cluster). The Connectors page opens.

3. Click Add connector. Complete the fields as described in adding a connector. Continue to the Test and verify page.

   

4. Click Download connector config file. The Download config file dialog opens.

   

5. Click the file type to download: Download .JSON or Download .properties.

   The file is downloaded into your downloads directory. When you are done downloading files, click x to close the download dialog.

   Example JSON connector config file:

   {
    "name": "datagen-users",
    "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "kafka.topic": "users",
    "max.interval": "990",
    "iterations": "1000000000",
    "quickstart": "users"
   }
   

   Example properties connector config file:

   name = datagen-users
   connector.class = io.confluent.kafka.connect.datagen.DatagenConnector
   key.converter = org.apache.kafka.connect.storage.StringConverter
   kafka.topic = users
   max.interval = 990
   iterations = 1000000000
   quickstart = users
   

6. Click Launch to launch the connector.

Upload a connector configuration file to add a connector

Upload a connector configuration file to add a connector. You can upload a connector config file from either the Browse connectors page when adding a connector, or directly from the Connectors page.

Supported formats for upload:

• .json file
• .properties file

1. Select a cluster from the navigation bar and click the Connect menu. The All Connect Clusters page opens.

2. In the Cluster name column, click the connect-default link (or the link for your Connect cluster). The Connectors page opens.

   

   Upload config from Connectors page

3. Click + Upload connector config file.

4. In the Downloads directory (or navigate to the directory where your config resides), select the configuration file you want to upload and click Open.

   • The Connector UI fields are populated with the entries in the configuration file.
   • If an error is displayed, make sure the file has valid JSON if applicable. Ensure the JSON or properties file has valid properties for the type of connector.

Edit a connector

Pause the connector before making configuration changes.

1. Select a cluster from the navigation bar and click the Connect menu. The All Connect Clusters page opens.
2. In the Cluster name column, click the connect-default link (or the link for your Connect cluster). The Connectors page opens.
3. Click the Name column for the connector.
4. Click the Settings tab. To view a summary, click the Overview tab.
5. Make the changes you want to the connector fields. To get more information about a field, hover over the info (i) icon. Connector class and Name cannot be changed.
6. Click Continue.
7. (Optional) Click Download connector config file. See Download a connector configuration file for details.
8. Review the information and click Launch.

Pause and resume a running connector

Pause a running connector if you need to troubleshoot or make configuration changes. You can pause or resume a connector from either the Overview or the Settings pages.

The Overview and Connectors pages display Paused in the Connectors summary panel and the Status column for a paused connector. See the following example from the Connectors page:

1. Select a cluster from the navigation bar and click the Connect menu. The All Connect Clusters page opens.
2. In the Cluster name column, click the connect-default link (or the link for your Connect cluster). The Connectors page opens.
3. Select a running connector, the select Pause. To allow the connector to run again, click Resume .

Restart a connector

Restart a connector if you need to reboot the connector and all its tasks. You can restart the connector by clicking Restart on the Overview page.

1. Select a cluster from the navigation bar and click the Connect menu. The All Connect Clusters page opens.
2. In the Cluster name column, click the connect-default link (or the link for your Connect cluster). The Connectors page opens.
3. Select a connector to restart and click Restart to restart the connector.
4. Verify the tasks have restarted and transition to the Runnning state.

Restart failed components

In the event that some or all of a connector’s tasks have failed, and/or the connector has failed, Control Center will display a message in the Overview page. If there is a failure, Control Center will display a message notifying one of the following:

• Tasks have failed. To restart the failed tasks, click Restart Failed Tasks to reboot only the failed tasks as shown in the following example:

  

• The connector has failed. To restart only the failed connector, click Restart Failed Connector.

• The connector and tasks have failed. To restart the failed connector and tasks, click Restart Failed Connector And Tasks.

Be sure to verify the failed components have restarted and transitioned to the Running state.

Delete a connector

Delete a connector that is no longer needed. You can pause a connector before deleting it. Deleting a connector stops a connector if it is running.

To delete a connector

1. Select a cluster from the navigation bar and click the Connect menu. The All Connect Clusters page opens.

   

2. In the Cluster name column, click the connect-default link. The Connectors page opens.

3. Click the Name column for the connector. The Overview page opens.

   

4. Click the trash icon. You are prompted to confirm the deletion.

5. Click Confirm.