Confluent Cloud API and Confluent CLI for Custom Connectors

See the following sections for information about using the Custom Connector API and CLI.

Custom Connector API

The Confluent Cloud API allows you to interact with your custom connectors using the Confluent Cloud API. For more information, see the following resources:

Custom Connector CLI

Custom Connector CLI commands allow you to manage your custom connectors using the Confluent CLI. The following examples show commands for common custom connector tasks.

Create or update a custom connector

The following examples show how to create a connector and update a connector configuration using the Confluent CLI.

Custom connector configuration payload

There are a few configuration properties for custom connectors that are not required for fully-managed connector configurations. Note the following configuration property differences in the configuration payload when creating or updating custom connectors using the Confluent Cloud API.

  • confluent.connector.type (string): This property defines the type of connector. The property accepts two values: CUSTOM or MANAGED. The default is MANAGED. If creating a custom connector, this property is required to be set to CUSTOM.
  • confluent.custom.plugin.id (string): This property provides the plugin ID for the custom connector. This is a unique ID assigned to the custom connector plugin uploaded to Confluent Cloud. This property is required for custom connectors. To get the plugin ID for a custom connector, see View a custom connector plugin ID.
  • confluent.custom.connection.endpoints (string): This property provides an allowed static endpoint (or endpoints) for the custom connector to use when securely attaching to a sink or source data system. This is optional for a custom connector configuration. For more information and endpoint requirements, see egress endpoints.
  • confluent.custom.schema.registry.auto (string): When set to TRUE, this property automatically adds the required Schema Registry properties to the custom connector configuration, if Schema Registry is enabled for the Confluent Cloud cluster. Defaults to FALSE. For more information about Schema Registry integration with custom connectors, see Schema Registry integration.

Note

kafka.auth.mode: The current mode for authentication defaults to API key and secret. You do not need to set this configuration property.

Create a custom connector

Use the following command to create a custom connector.

Command syntax:

confluent connect cluster create [flags]

For example:

confluent connect cluster create --config-file connector-config.json --cluster lkc-abcd1e --environment env-a12b34

Example output:

+------+---------------------+
| ID   | clcc-wzxp69         |
| Name | my-custom-connector |
+------+---------------------+

Note that the ID of a Custom Connector starts with a prefix of clcc and the ID of a Managed Connector starts with a prefix of lcc.

The JSON payload file consists of the following configuration properties. Note that the connector name used in both instances of "name" in the payload must be consistent.

{
   "name": "my-custom-connector",
   "config": {
         "name": "my-custom-connector",
         "kafka.api.key": "********",
         "kafka.api.secret": "********",
         "confluent.connector.type": "CUSTOM",
         "confluent.custom.plugin.id": "custom-plugin-l65664",
         "tasks.max": "1",
         "interval.ms": "10000",
         "kafka.topic": "my-kafka-topic"
         }
}

Update a custom connector configuration

Use the following command to update a custom connector configuration. You use a JSON payload file that contains all the configuration properties used to create the original connector, with any changes needed for the update. In the example JSON used in this example, the tasks.max is updated from 1 to 2.

Command syntax:

confluent connect cluster update <id> [flags]

For example:

confluent connect cluster clcc-wzxp69 --config-file connector-config.json --cluster lkc-abcd1e --environment env-a12b34

Example output:

Updated connector "clcc-wzxp69"

The JSON payload file consists of the following configuration properties. Note that the connector name used in both instances of "name" in the payload must be consistent.

{
   "name": "my-custom-connector",
   "config": {
         "name": "my-custom-connector",
         "kafka.api.key": "********",
         "kafka.api.secret": "********",
         "confluent.connector.type": "CUSTOM",
         "confluent.custom.plugin.id": "custom-plugin-l65664",
         "tasks.max": "2",
         "interval.ms": "10000",
         "kafka.topic": "my-kafka-topic"
         }
}

Delete a custom connector

Use the following command to delete a connector.

Command syntax:

confluent connect cluster delete <id> [flags]

For example:

confluent connect cluster delete clcc-wzxp69 --cluster lkc-abcd1e

Example output:

Are you sure you want to delete connector "clcc-wzxp69"?
To confirm, type "my-custom-connector". To cancel, press Ctrl-C: my-custom-connector
Deleted connector "clcc-wzxp69".

Describe a custom connector

Use the following command to get connector details.

Command syntax:

confluent connect cluster describe <id> [flags]

For example:

confluent connect cluster describe clcc-wzxp69 --cluster lkc-abcd1e

Example output:

  Connector Details
+--------+---------------------+
| ID     | clcc-wzxp69         |
| Name   | my-custom-connector |
| Status | RUNNING             |
| Type   | source              |
+--------+---------------------+


Task Level Details
  Task ID |  State
----------+----------
        0 | RUNNING


Configuration Details
              Config             |                          Value
---------------------------------+----------------------------------------------------------
  cloud.environment              | prod
  cloud.provider                 | aws
  confluent.custom.plugin.id     | custom-plugin-epp0ye
  connector.class                | io.confluent.kafka.connect.datagen.DatagenConnector
  iterations                     |                                                10000000
  kafka.api.key                  | ****************
  kafka.api.secret               | ****************
  kafka.auth.mode                | KAFKA_API_KEY
  kafka.endpoint                 | SASL_SSL://pkc-abcd5.us-west-2.aws.confluent.cloud:9092
  kafka.region                   | us-west-2
  kafka.topic                    | pageviews
  key.converter                  | org.apache.kafka.connect.storage.StringConverter
  max.interval                   |                                                     100
  name                           | custom-datagen_0
  quickstart                     | pageviews
  tasks.max                      |                                                       1
  value.converter                | org.apache.kafka.connect.json.JsonConverter
  value.converter.schemas.enable | false

Get a list of custom connectors

Use the following command to get a list of connectors.

Command syntax:

confluent connect cluster list [flags]

For example:

confluent connect cluster list --cluster lkc-abcd1e

Example output:

      ID      |       Name          | Status  |  Type  | Trace
--------------+---------------------+---------+--------+--------
  clcc-wzxp69 | my-custom-connector | RUNNING | source |

Note that the ID of a Custom Connector starts with a prefix of clcc and the ID of a Managed Connector starts with a prefix of lcc.

Pause a custom connector

Use the following command to pause a connector.

Command syntax:

confluent connect cluster pause <id1> <id2> ...<idN> [flags]

For example:

confluent connect cluster pause clcc-wzxp69 --cluster lkc-abcd1e

Example output:

Paused connector "clcc-wzxp69".

Resume a paused custom connector

Use the following command to resume a paused connector.

Command syntax:

confluent connect cluster resume <id1> <id2> ...<idN> [flags]

For example:

confluent connect cluster resume clcc-wzxp69 --cluster lkc-abcd1e

Example output:

Resumed connector "clcc-wzxp69".

Custom Connector Plugin CLI

Custom Connector plugin CLI commands allow you to manage your custom connector plugins using the Confluent CLI. The following examples show commands for common custom connector plugin tasks.

List custom plugins

The following example shows how to list custom plugins using the CLI.

Command syntax:

confluent connect custom-plugin list

Example output:

      ID     |       Name
-------------+-------------------
  ccp-abcdef | custom-datagen-1

Describe a custom plugin

The following examples show how to describe a custom plugin using the CLI.

Command syntax:

confluent connect custom-plugin describe <plugin-ID> [flags]

For example:

confluent connect custom-plugin describe ccp-abcdef

Example output:

 +----------------------+-----------------------------------------------------+
| ID                   | ccp-abcdef                                          |
| Name                 | custom-datagen-2                                    |
| Description          |                                                     |
| Documentation Link   |                                                     |
| Connector Class      | io.confluent.kafka.connect.datagen.DatagenConnector |
| Connector Type       | SOURCE                                              |
| Sensitive Properties | foo,bar                                             |
+----------------------+-----------------------------------------------------+

Create a custom plugin

The following examples show how to create a custom plugin using the CLI. The following are required command option:

  • --plugin-file: The path to the locally stored ZIP or JAR file to be uploaded.
  • --connector-class: Java class or alias for connector. You can get connector class from connector documentation provided by developer.
  • --connector-type: SOURCE or SINK.

Optional command options include the following:

  • --description: A meaningful description for the plugin. Note that if adding a description with more than one word, add quotes to the start and end. For example --description "A test connector for creating mock source data."
  • --documentation-link: You can include a URL for documentation users need to reference for the connector. This URL will be displayed for users configuring a connector using the plugin.
  • --sensitive_properties: A sensitive property is a connector configuration property that must be hidden after a user enters the property value when setting up connector. For more information about sensitive properties, see Uploading and launching the connector.

Command syntax:

confluent connect custom-plugin create <plugin-name> [flags]

For example:

confluent connect custom-plugin create custom-datagen-2 --plugin-file /Users/<username>/Downloads/confluentinc-kafka-connect-datagen-0.6.0.zip --connector-class io.confluent.kafka.connect.datagen.DatagenConnector --connector-type SOURCE --sensitive-properties foo,bar

Example output:

+------+------------------+
| ID   | ccp-ghijkl       |
| Name | custom-datagen-2 |
+------+------------------+

Update a custom plugin

The following examples show how to update a custom plugin using the CLI.

Command syntax:

confluent connect custom-plugin update <plugin-ID> [flags]

For example, to change the plugin name:

confluent connect custom-plugin update ccp-4xqy7x --name custom-datagen-0

Example output:

Updated custom connector plugin "ccp-ghijkl"

Delete a custom plugin

The following examples show how to delete a custom plugin using the CLI.

Command syntax:

confluent connect custom-plugin delete <id-1> [id-2] ... [id-n] [flags]

For example:

confluent connect custom-plugin delete ccp-ghijkl

Example output:

Are you sure you want to delete custom connector plugin "ccp-ghijkl"?
To confirm, type "custom-datagen-0". To cancel, press Ctrl-C: custom-datagen-0
Deleted custom connector plugin "ccp-ghijkl".

Unsupported connector CLI commands

The following commands are not supported by Custom Connectors. These CLI commands are applicable only to fully-managed connectors.

Command reference

For the full Confluent CLI Command Reference for fully-managed and custom connectors, see confluent connect.