Confluent Developer Marketplace Get started
  1. Home
  2. Cloud
  3. Unified Stream Manager Overview
  4. Register a Connect Cluster Overview

Confluent Platform Kafka as the Metadata Cluster

This page covers registering a self-managed Connect cluster that produces to and consumes from a Confluent Platform Kafka cluster, which serves as its metadata cluster, using Unified Stream Manager (USM). For the Confluent Cloud Kafka case instead, see Confluent Cloud Kafka as the Metadata Cluster.

Before you begin

Register a Connect cluster against Confluent Platform Kafka

To register a self-managed Connect cluster:

  1. Log in to your Confluent Platform cluster by using the Confluent Platform CLI.

  2. Get the Connect cluster ID by running the following command to list the available Kafka Connect clusters:

    confluent connect cluster list

  3. From the output, copy the ID of the Kafka Connect cluster that you want to register.

  4. In the Confluent Cloud Console, navigate to Environments and select your environment.

    ../../_images/usm-env.png

  5. In the navigation menu, click Clusters and select a Confluent Platform cluster.

    ../../_images/usm-cluster.png

  6. In the navigation menu, click Connectors.

    ../../_images/usm-connect-register.png

  7. Click Register Connect cluster.

    ../../_images/usm-connect-registerselect.png

  8. Paste the Kafka Connect cluster ID that you copied.

    ../../_images/usm-connector.png

  9. Click Finish.

Data from your connectors might take a few minutes to appear in the Confluent Cloud.

Alternatively, if connectors are already running but the cluster isn’t registered, a Register Connect cluster button is also available on the Connectors page.

To register a Confluent Platform Connect cluster, use the confluent usm connect-cluster register command.

confluent usm connect-cluster register <connect_ID> --kafka-cluster-id <cluster-ID> --cloud <cloud-type> --region <region-name>

For --cloud, specify aws or azure, and for --region, specify a region from Configure AWS Private Networking for USM Agents (AWS) or Configure Azure Private Networking for USM Agents (Azure).

For example, to register a Kafka cluster with ID 4k0R9d1GTS5tI9f4Y2xZ0Q and connector ID connect-group-xyz123 in AWS region us-east-1:

confluent usm connect-cluster register connect-group-xyz123 --kafka-cluster-id 4k0R9d1GTS5tI9f4Y2xZ0Q --cloud aws --region us-east-1

The following is an example of the output:

+--------------------------------+--------------------------+
| ID                             | usmcc-global-ghijkl67890 |
| Confluent Platform Connect     | connect-group-xyz123     |
| Cluster                        |                          |
| USM Kafka Cluster ID           | usmkc-global-abcdef12356 |
| Confluent Platform Kafka       | 4k0R9d1GTS5tI9f4Y2xZ0Q   |
| Cluster ID                     |                          |
| Cloud                          | AWS                      |
| Region                         | us-east-1                |
| Environment                    | env-stgcqqd737           |
+--------------------------------+--------------------------+

To register a Connect cluster in Azure region centralus:

confluent usm connect-cluster register connect-group-xyz123 --kafka-cluster-id 9f4Y2xZ0Q4k0R9d1GTS5tI --cloud azure --region centralus

The following is an example of the output:

+--------------------------------+--------------------------+
| ID                             | usmcc-global-lkjihg09876 |
| Confluent Platform Connect     | connect-group-xyz123     |
| Cluster                        |                          |
| USM Kafka Cluster ID           | usmkc-global-fedcba54321 |
| Confluent Platform Kafka       | 9f4Y2xZ0Q4k0R9d1GTS5tI   |
| Cluster ID                     |                          |
| Cloud                          | AZURE                    |
| Region                         | centralus                |
| Environment                    | env-stgcqqd737           |
+--------------------------------+--------------------------+

After registration, the Connectors page populates with all connectors that are associated with that cluster. For more information about monitoring these connectors, see Monitor Confluent Platform Connectors with USM.

For a complete reference of related commands, see the confluent usm CLI documentation.

Register an additional Connect cluster

To register an additional Connect cluster, do the following:

  1. Navigate to Connectors for your registered Confluent Platform cluster.

  2. Click Connector cluster list.

  3. Click Register Connect cluster.

  4. In your Confluent Platform environment, get the ID of the additional Connect cluster by running the following command:

    confluent connect cluster list

  5. Enter the Confluent Platform Connect cluster ID in the Confluent Cloud Console.

  6. Click Finish.

Configure Connect workers for metadata sync

After registering your cluster, you must ensure your Connect workers can securely send metadata to Confluent Cloud. The required actions depend on your Confluent Platform deployment method.

For Confluent for Kubernetes or Ansible Playbooks for Confluent Platform deployments

If you deployed Confluent Platform using Confluent for Kubernetes or Ansible Playbooks, no additional configuration is needed. These tools automatically manage the settings required for metadata synchronization. After the registration is complete, your connector metadata begins to appear in Confluent Cloud.

Note

Confluent Platform Connect versions 7.9.6, 7.9.7, 7.9.8, 8.1.2, 8.1.3, 8.1.4, 8.2.0, 8.2.1, 8.2.2, and 8.3.0 have a known issue that prevents Confluent for Kubernetes and Ansible Playbooks for Confluent Platform deployments from emitting the metrics required for USM. If connector metrics do not appear in Confluent Cloud after registration, see Issue: Connect metrics are missing from USM to apply the workaround.

For manual or other deployments

If you installed or manage Confluent Platform manually, you must update the properties file for each Kafka Connect worker in the registered cluster.

  1. Open the properties file for each Kafka Connect worker in the cluster that you registered.

  2. Confirm that the bootstrap.servers property correctly points to the Kafka cluster that the Connect workers use to store connector metadata.

  3. Add the following metadata properties to the worker configuration file to enable the metadata events service.

    confluent.connect.metadata.events.enabled=true
confluent.connect.metadata.events.client.base.url=https://usm-agent.connect.svc.cluster.local:10000
confluent.connect.metadata.events.ssl.key.password=${file:/mnt/sslcerts/tls-connect/jksPassword.txt:jksPassword}
confluent.connect.metadata.events.ssl.keystore.location=/mnt/sslcerts/tls-connect/keystore.p12
confluent.connect.metadata.events.ssl.keystore.password=${file:/mnt/sslcerts/tls-connect/jksPassword.txt:jksPassword}
confluent.connect.metadata.events.ssl.protocol=TLSv1.2
confluent.connect.metadata.events.ssl.truststore.location=/mnt/sslcerts/tls-connect/truststore.p12
confluent.connect.metadata.events.ssl.truststore.password=${file:/mnt/sslcerts/tls-connect/jksPassword.txt:jksPassword}

  4. Save your changes.

  5. Perform a rolling restart of your Connect workers. After they restart, the workers send metadata to Confluent Cloud.

Related content