Configure Kafka Connect for Confluent Platform Using Confluent for Kubernetes Blueprints

Kafka Connect is a tool for streaming data between Kafka and other data systems. It uses connectors to stream data into or out of Kafka. Connectors in Connect define where data should be copied to and from.

Confluent for Kubernetes (CFK) Blueprints provides the platform admins the capability to install and manage connectors via the ConfluentPlatformBlueprint and the Confluent cluster custom resource (CR), such as to:

At the deployment level, the ConnectCluster CR should specify which connectors should be installed for the specific Confluent Platform deployment.

Adding connectors to an existing Connect cluster does not require the Connect cluster to be restarted.

Configure how to load connector plugins

Connector plugins are the binaries or JARs that implement the classes/abstractions of connectors. Connector plugins are installed in Connect workers.

When configuring Connect, you need to specify how to install connector plugins for the Connect deployment.

we should also talk about how connectCluster can configure to load the specific connectors supported by blueprint. The connectCluster Class references the supported connector via name.

After you install the connector plugins as described in this topic, see Manage Connectors for Confluent Platform Using Confluent for Kubernetes Blueprints for declaratively managing connectors using the Connector custom resource definition (CRD).

CFK Blueprints can download and install connector plugins/JARs from Confluent Hub or a custom artifacts location URL.

CFK Blueprints does not support downloading connectors from two different location types. You can set a ConnectClusterClass CR to download either from Confluent Hub or from a custom location.

To store the connector plugins, you need to have a specified size of node volume available. The default size is 4 GB, but you can specify a different size in the ConnectClusterClass CR using storageLimit.

Provide the download info in the ConnectClusterClass CR:

kind: ConnectClusterClass
spec:
  provisioner:
    cfk:
      connectorPlugins:
        confluentHubPlugins:     --- [1]
          - name:                --- [2]
            owner:               --- [3]
            version:             --- [4]
        urlPlugins:              --- [5]
          - name:                --- [6]
            archivePath:         --- [7]
            checksum:            --- [8]
      storageLimit:              --- [9]

  • Only provide confluentHubPlugins ([1]) or urlPlugins ([2]) and not both.

  • [1] Provide an array of plugins to be downloaded from Confluent Hub.

  • [2] Required. Specify a list of the connector plugins allowed to be installed on the Connect cluster.

    The connectCluster CR references these names to select the connectors to install in spec.build.static.plugins.

  • [3] Required. The individual or organization that provides the plugin, for example, confluentinc.

  • [4] Required. The version of this plugin. Set to the version of the plugin or latest.

  • [5] Provide an array of plugins to be downloaded from a custom location.

  • [6] Required. Specify a list of the connector plugins allowed to be installed on the Connect cluster. The connectCluster CR references these names to select the connectors to install in spec.build.static.plugins.

  • [7] Required. The archive path of the zip file that contains this plugin.

  • [8] Required. Defines the sha512sum checksum of the plugin’s remote file. It is used to verify the remote file after download.

  • [9] The maximum amount of node volume that can be used to store the connector plugins. The default value is 4 GB.