Custom Connector for Confluent Cloud Quick Start

Deploy a custom connector plugin in Confluent Cloud to integrate external systems using your own code. This quick start walks you through getting, packaging, and launching a connector.

Deployment steps:

1. Get a connector

2. Package a custom connector

3. Upload and launch a custom connector

Prerequisites

Before you begin, ensure you review limitations and support for custom connectors.

Get a connector

To integrate your data system with Confluent Cloud, choose one of the following options to get a custom connector:

• Download a pre-built connector

• Modify an open-source connector

• Build a custom connector from scratch

• Confluent Professional Services

• Confluent partner connectors

Download a pre-built connector

If an existing connector supports your target data system, select an option below to download a pre-packaged connector plugin archive or retrieve open-source code:

Confluent Marketplace

Search Confluent Marketplace to find pre-built, self-managed connectors for your data system.

1. Go to Confluent Marketplace and enter the name of the connector or data system in the search field.

2. Apply search filters, such as deployment and connector type, to refine your search.

3. Click the connector name to view its details. The supported deployment cards are displayed. For example, the Oracle CDC Source Connector supports self-hosted, Confluent Platform, and Confluent Cloud deployment options.

   

   Connector details

4. In the Confluent Cloud card, click Download, and then select your preferred version.

   • Connectors downloaded from Confluent Marketplace use the required file structure and can be uploaded directly to Confluent Cloud.

   • To modify open-source connector code before uploading, see Modify an open-source connector.

5. Upload the connector plugin archive to Confluent Cloud.

GitHub

Build custom connectors using open-source Apache Kafka® projects hosted on GitHub.

1. Go to GitHub and search for the preferred connector repository.

2. Download the project ZIP file.

   

   GitHub download

3. Extract the downloaded source files.

4. Package the connector plugin archive. Use the Maven Packaging Plugin to automatically format your connector files into the required archive structure.

5. Upload the connector plugin archive to Confluent Cloud.

Note

Verify that the repository includes an acceptable open-source license before building and deploying it to production.

Modify an open-source connector

If available connectors do not support your specific data requirements, you can customize an existing open-source connector. Source code is available on Confluent Marketplace and GitHub. Clone the project, make your modifications, and package the code into a ZIP or JAR archive to upload to Confluent Cloud.

Confluent Marketplace

1. Search Confluent Marketplace for the connector you want to modify.

2. Filter by License > Free to view all open-source connectors.

3. Click the connector to open its details page.

4. In the Confluent Cloud section, click View instructions to open its GitHub repository.

   

   Viewing Connector details on GitHub

5. Download the project ZIP file or clone the repository locally.

6. Modify the source code to meet your requirements. For more information, see Build a custom connector from scratch.

7. Package the connector plugin archive. Use the Maven Packaging Plugin to automatically format your modified files into the required structure.

8. Upload the custom connector plugin archive to Confluent Cloud.

Note

Verify that the repository uses an appropriate open-source license before customizing and deploying the code.

GitHub

1. Search GitHub for the connector repository you want to modify.

2. Download the project ZIP file or clone the project locally.

   

   GitHub download

3. Extract the downloaded source files.

4. Modify the source code to meet your requirements. For more information, see Build a custom connector from scratch.

5. Package the connector plugin archive. Use the Maven Packaging Plugin to automatically format your modified files into the required structure.

6. Upload the custom connector plugin archive to Confluent Cloud.

Note

Verify that the repository uses an appropriate open-source license before customizing and deploying the code.

Build a custom connector from scratch

To build a custom connector from scratch, use Maven and the Kafka Connect Archetype Quickstart to generate a project skeleton.

For architecture, API specifications, and core development principles, see the Connector Developer Guide.

Additional resources for building custom connectors

• Deep Dive into Configuration Handling

• 4 Steps to Creating Apache Kafka Connectors with the Kafka Connect API

• From Zero to Hero with Kafka Connect

• Kafka Connect Source Connectors: A Detailed Guide

After developing your connector code, package the connector files into an archive before uploading to Confluent Cloud.

Confluent Professional Services

Confluent Professional Services can partner with your team to provide expert advice and guidance for building custom connectors. Contact your Confluent account team for more information.

Confluent partner connectors

Confluent partners offer specialized connectors for other data systems. Contact your Confluent account team to check if a partner-developed connector is available for your use case.

Package a custom connector

Before uploading a custom connector to Confluent Cloud, you must package your connector plugin files into a .zip or .jar archive.

Prerequisites

Before you begin, ensure you have:

• Permissions: An administrator RBAC role to upload archives. For details, see RBAC role mappings.

• Size limit: Archive files cannot exceed 250 MB.

• Scope: Custom connector plugins are deployed at the organization level and are accessible across clusters.

• ZIP format: If uploading a .zip file, the archive must follow the Confluent-supported file structure.

Use the Maven Packaging Plugin to automatically format and package your connector files into the required structure.

For manual packaging rules, manifest file specifications, and sample structures, see the Confluent Marketplace Component Archive Specification.

Connector plugin archive structure

A connector plugin archive contains the following files and directories:

• manifest.json: Contains mandatory archive metadata and specifications.

• assets/: Contains icons and logo assets for display in the UI.

• doc/: Contains documentation, licensing information, and notice files.

• etc/: Contains sample connector configuration properties.

• lib/: Contains all Java dependencies and JAR files needed to run the connector.

Note

• The lib/ directory must include all JAR dependencies required by the connector, including external client libraries, custom Single Message Transformations (SMTs), and non-Kafka JAR files.

• Standard SMTs from the Confluent Transforms archive are pre-installed and do not need to be included in your package.

The following examples show the archive structure and files.

confluentinc-kafka-connect-datagen-0.6.0/
assets/
     confluent.png
doc/
     LICENSE
     README.md
     licenses
     licenses.html
     ...
     notices
etc/
     connector_campaign_finance.config
     connector_credit_cards.config
     connector_custom.config
     connector_inventory.config
     ...
lib/
     automaton-1.11-8.jar
     avro-1.11.0.jar
     avro-random-generator-0.4.1.jar
     checker-qual-3.5.0.jar
     common-utils-7.2.0.jar
     commons-compress-1.21.jar
     error_prone_annotations-2.3.4.jar
     ...
     swagger-annotations-2.1.10.jar
manifest.json

manifest.json file

The manifest.json file defines essential metadata for your custom connector plugin and can include many fields.

The following table describes key manifest fields:

Field	Definition	Required/Optional
version	Sets the plugin version string.	Required
documentation_url	URL pointing to plugin documentation or repository.	Optional
kafka_version	The Kafka version the connector plugin compiles against.	Optional
java_version	The Java runtime version the connector plugin targets.	Optional

Example file:

{
  "name": "kafka-connect-my-custom-connector",
  "version": "1.0.0",
  "documentation_url": "https://github.com/myorg/kafka-connect-my-custom-connector/blob/main/README.md",
  "kafka_version": "3.6.0",
  "java_version": "17",
  ...
}

Upload and launch a custom connector

After you package your custom connector plugin, upload the packaged .zip or .jar file to Confluent Cloud. When the upload finishes, users across your organization can configure and launch instances of the connector.

Prerequisites

Before you begin, ensure you have:

• Cluster access: Access to a Confluent Cloud cluster in a supported region.

• Permissions: An administrator RBAC role with permissions to upload custom connectors.

• File format and structure: Archives in .zip or .jar format not exceeding 250 MB. ZIP files must follow the required file structure.

You can upload a custom connector plugin archive from either the Artifacts page or the Connector Plugins page.

Artifacts

1. In your Confluent Cloud environment, select Artifacts, and then select the Custom connectors tab.

2. Click Upload Plugin Artifact.

   

   Upload connector plugin on Artifacts page

3. On the Upload custom connector plugin page, select Create new plugin.

4. Enter a Plugin name.

5. Enter an optional Description.

6. Select the Cloud provider.

7. Click Upload JAR file and select your connector archive. Confluent Cloud scans the uploaded archive and extracts the connector classes and configurations.

   Note

   Archives must be in .zip or .jar format and cannot exceed 250 MB.

8. Click Continue.

Confluent Cloud adds the connector plugin to the Artifacts page. To add a version to an existing plugin, see Add a custom connector plugin version.

Connector Plugins

1. Navigate to the Connectors page in your Confluent Cloud cluster.

2. Optional. If existing connectors are listed, click Add Connector.

3. On the Connector Plugins page, click Add custom connector plugin.

   

   Connector Plugins page

4. Enter the required details:

   • Enter a Plugin name.

   • Enter an optional Description.

   • Select the Cloud provider.

   • Click Upload JAR file and select your connector archive. Confluent Cloud scans the uploaded archive and extracts the connector classes and configurations.

     Note

     Archives must be in .zip or .jar format and cannot exceed 250 MB.

5. Click Continue.

Confluent Cloud adds the connector plugin to the Connector Plugins page.

Configure and launch a connector

Prerequisites

Before configuring and launching a custom connector, ensure you have:

• Access to a Confluent Cloud cluster in a supported region.

• Authentication credentials for the target data system.

• Required egress network endpoints configured for the target system.

• Schema Registry enabled on your cluster if using Schema Registry-based formats, such as Avro, JSON Schema, or Protobuf. For details, see Schema Registry integration.

• The list of required connector configuration properties.

• The Confluent CLI installed and authenticated to your cluster. See Install the Confluent CLI.

Note

Many connectors require target Kafka topics to exist before launch. Check your connector documentation for specific requirements.

On the Connector Plugins page, filter plugins by Custom, and then select the plugin you want to configure, or hover over the connector card and click Get Started.

The add connector configuration step is displayed.

Tip

Review the plugin documentation for required configuration properties and deployment details.

Step 1: Add connector details

In the Connector details step, select plugin details, such as version, type, Kafka, and JDK versions.

Step 2: Add credentials

In the Kafka access step, create or select an API key. You can choose one of the following options:

• My Account to generate an API key and secret that your connector can use to communicate with your Kafka cluster.

• Service Account to select or create a service account for your connector to communicate with your Kafka cluster. This option is recommended for production use.

• Use an existing API key to enter an API key and secret pair you have stored.

Step 3: Add configuration properties

During the Configuration step, Confluent Cloud automatically populates the configuration properties extracted from your connector plugin. Enter the required values for your target deployment. The displayed properties depend on the selected plugin version and connector.class.

Manage Schema Registry settings

To manage schema serialization and deserialization formats, configure the following properties:

• Key converter class

• Value converter class

• Schema Registry auto mode

For more information, see Schema Registry integration.

Configure advanced properties

Expand Show advanced configurations to configure the following:

• Optional connector properties: Configure parameters defined with low or medium importance in the connector file that are not required for basic setup.

• Additional properties: Add custom properties to override default connector behavior. For supported configuration overrides, see Override configuration properties.

• Single Message Transforms (SMTs): Add transformation rules to change message keys, values, or headers. For details, see Single Message Transformations.

Step 4: Add networking endpoints

In the Networking step, specify the connection endpoints required for the connector to reach your source or sink system.

Endpoints must use a fully qualified domain name (FQDN). Protocol specification is optional and defaults to TCP. For details, see Endpoint requirements.

Use the following syntax pattern:

<HOSTNAME_PATTERN>:<PORTS_LIST>[:<PROTOCOL>]

Examples:

• Amazon S3: s3.us-west-2.amazonaws.com:443

• Amazon RDS: mydatabase.abc123ecs2.us-west-2.rds.amazonaws.com:1433:TCP

• Wildcard pattern: *database.abc123ecs2.us-west-2.rds.amazonaws.com:1433:TCP

Note

Confluent automatically applies a single-level wildcard to the leftmost label of a domain. For example, specifying s3.us-west-2.amazonaws.com:443 implicitly permits access to *.s3.us-west-2.amazonaws.com:443.

Entering subdomains or multi-level wildcards, such as *.s3.us-west-2.amazonaws.com:443, as input is not supported.

Endpoint requirements

• Separate multiple endpoints using a semicolon.

• Separate the hostname, ports, and protocols with colons.

• Separate multiple ports using a comma.

• Do not include HTTP or HTTPS with the endpoint.

• The protocol is optional and defaults to TCP. Using ANY is not allowed.

Note the following wildcard requirements:

• Only one wildcard is allowed per endpoint.

• The wildcard is only allowed in the leftmost element of the endpoint entry.

• Using a wildcard for a subdomain is not allowed.

The endpoint is checked for valid characters. Hostnames are checked for valid patterns (excluding a wildcard) and against the Web Risk API. For more information, see Web Risk.

Important

Confluent routinely scans uploaded custom connectors to detect security risks and interactions with malicious endpoints. If malicious activity is detected, Confluent reserves the right to delete the connector immediately.

Step 5: Configure sizing

In the Sizing step, enter the number of Tasks to allocate for the connector.

• Minimum tasks: Enter 1. This is recommended for testing or low-volume workloads.

• Scaling for throughput: Increase the task count to process data in parallel across multiple partitions or files.

Note

Connector memory of 2 GB is shared across all allocated tasks. Increasing the number of tasks increases throughput and hourly costs, but splits the shared memory among active tasks.

Step 6: Review and launch

In the Review and launch step, review the provisioned details and select Continue. Connector configurations marked with an asterisk (*) cannot be changed after you launch your connector.

The connector’s status changes to running after provisioning completes.

Connector provisioning

Note

If the connector remains in the Provisioning state for an extended period, verify that all required properties are specified correctly without typos or syntax errors.

For details on updating configuration settings, see Modify a custom connector configuration.

Next steps

• To search, monitor, reconfigure, or delete a connector or plugin after launch, see Manage Custom Connectors for Confluent Cloud.

• To automate custom connector or plugin tasks from the command line, see Confluent Cloud API and Confluent CLI for Custom Connectors.

• For custom connector and custom connector plugin API information and examples, see the Confluent Cloud API for Connect Usage Examples.