# Custom Connector for Confluent Cloud Quick Start Deploy a custom connector plugin in Confluent Cloud to integrate external systems using your own code. This quick start walks you through getting, packaging, and launching a connector. Deployment steps: 1. [Get a connector](#cc-byoc-getting-connector) 2. [Package a custom connector](#cc-byoc-packaging) 3. [Upload and launch a custom connector](#cc-byoc-upload-provision) ## Prerequisites Before you begin, ensure you review [limitations and support](custom-connector-fands.md#cc-bring-your-connector-fands) for custom connectors. ## Get a connector To integrate your data system with Confluent Cloud, choose one of the following options to get a custom connector: * [Download a pre-built connector](#cc-byoc-get-hub) * [Modify an open-source connector](#cc-byoc-get-open-source) * [Build a custom connector from scratch](#cc-byoc-get-create-new) * [Confluent Professional Services](#cc-byoc-get-prof-services) * [Confluent partner connectors](#cc-byoc-get-partner) ### Download a pre-built connector If an existing connector supports your target data system, select an option below to download a pre-packaged connector plugin archive or retrieve open-source code: ### Confluent Marketplace Search Confluent Marketplace to find pre-built, self-managed connectors for your data system. 1. Go to [Confluent Marketplace](https://www.confluent.io/hub/) and enter the name of the connector or data system in the search field. 2. Apply search filters, such as deployment and connector type, to refine your search. 3. Click the connector name to view its details. The supported deployment cards are displayed. For example, the **Oracle CDC Source Connector** supports self-hosted, Confluent Platform, and Confluent Cloud deployment options. ![Connector details page](images/ccloud-byoc-hub-oracle-cdc.png) 4. In the **Confluent Cloud** card, click **Download**, and then select your preferred version. * Connectors downloaded from Confluent Marketplace use the [required file structure](#cc-byoc-packaging) and can be uploaded directly to Confluent Cloud. * To modify open-source connector code before uploading, see [Modify an open-source connector](#cc-byoc-get-open-source). 5. [Upload the connector plugin archive](#cc-byoc-upload-provision) to Confluent Cloud. ### GitHub Build custom connectors using open-source Apache Kafka® projects hosted on GitHub. 1. Go to [GitHub](https://github.com/) and search for the preferred connector repository. 2. Download the project ZIP file. ![Download GitHub connector source](images/ccloud-byoc-github-download-zip.png) 3. Extract the downloaded source files. 4. [Package the connector plugin archive](#cc-byoc-packaging). Use the [Maven Packaging Plugin](https://docs.confluent.io/kafka-connectors/self-managed/confluent-hub/component-archive.html#maven-packaging-plugin) to automatically format your connector files into the required archive structure. 5. [Upload the connector plugin archive](#cc-byoc-upload-provision) to Confluent Cloud. #### NOTE Verify that the repository includes an acceptable [open-source license](https://choosealicense.com/licenses/) before building and deploying it to production. ### Modify an open-source connector If available connectors do not support your specific data requirements, you can customize an existing open-source connector. Source code is available on Confluent Marketplace and GitHub. Clone the project, make your modifications, and [package the code](#cc-byoc-packaging) into a ZIP or JAR archive to upload to Confluent Cloud. ### Confluent Marketplace 1. Search [Confluent Marketplace](https://www.confluent.io/hub/) for the connector you want to modify. 2. Filter by **License** > **Free** to view all open-source connectors. 3. Click the connector to open its details page. 4. In the **Confluent Cloud** section, click **View instructions** to open its GitHub repository. ![Accessing Connector details on GitHub](images/ccloud-byoc-hub-view-instructions.png) 5. Download the project ZIP file or clone the repository locally. 6. Modify the source code to meet your requirements. For more information, see [Build a custom connector from scratch](#cc-byoc-get-create-new). 7. [Package the connector plugin archive](#cc-byoc-packaging). Use the [Maven Packaging Plugin](https://docs.confluent.io/kafka-connectors/self-managed/confluent-hub/component-archive.html#maven-packaging-plugin) to automatically format your modified files into the required structure. 8. [Upload the custom connector plugin archive](#cc-byoc-upload-provision) to Confluent Cloud. #### NOTE Verify that the repository uses an appropriate [open-source license](https://choosealicense.com/licenses/) before customizing and deploying the code. ### GitHub 1. Search [GitHub](https://github.com/) for the connector repository you want to modify. 2. Download the project ZIP file or clone the project locally. ![Download GitHub connector source](images/ccloud-byoc-github-download-zip.png) 3. Extract the downloaded source files. 4. Modify the source code to meet your requirements. For more information, see [Build a custom connector from scratch](#cc-byoc-get-create-new). 5. [Package the connector plugin archive](#cc-byoc-packaging). Use the [Maven Packaging Plugin](https://docs.confluent.io/kafka-connectors/self-managed/confluent-hub/component-archive.html#maven-packaging-plugin) to automatically format your modified files into the required structure. 6. [Upload the custom connector plugin archive](#cc-byoc-upload-provision) to Confluent Cloud. #### NOTE Verify that the repository uses an appropriate [open-source license](https://choosealicense.com/licenses/) before customizing and deploying the code. ### Build a custom connector from scratch To build a custom connector from scratch, use [Maven](https://maven.apache.org/) and the [Kafka Connect Archetype Quickstart](https://github.com/jcustenborder/kafka-connect-archtype) to generate a project skeleton. For architecture, API specifications, and core development principles, see the [Connector Developer Guide](https://docs.confluent.io/platform/current/connect/devguide.html). #### Additional resources for building custom connectors * [Deep Dive into Configuration Handling](https://www.confluent.io/blog/write-a-kafka-connect-connector-with-configuration-handling/) * [4 Steps to Creating Apache Kafka Connectors with the Kafka Connect API](https://www.confluent.io/blog/create-dynamic-kafka-connect-source-connectors/) * [From Zero to Hero with Kafka Connect](https://www.confluent.io/kafka-summit-lon19/from-zero-to-hero-with-kafka-connect/) * [Kafka Connect Source Connectors: A Detailed Guide](https://opencredo.com/blogs/kafka-connect-source-connectors-a-detailed-guide-to-connecting-to-what-you-love/) After developing your connector code, [package the connector files into an archive](#cc-byoc-packaging) before uploading to Confluent Cloud. ### Confluent Professional Services Confluent Professional Services can partner with your team to provide expert advice and guidance for building custom connectors. Contact your Confluent account team for more information. ### Confluent partner connectors Confluent partners offer specialized connectors for other data systems. Contact your Confluent account team to check if a partner-developed connector is available for your use case. ## Package a custom connector Before uploading a custom connector to Confluent Cloud, you must package your connector plugin files into a `.zip` or `.jar` archive. ### Prerequisites Before you begin, ensure you have: * **Permissions:** An administrator RBAC role to upload archives. For details, see [RBAC role mappings](../managed-connector-rbac.md#managed-connector-rbac-role-mappings). * **Size limit:** Archive files cannot exceed 250 MB. * **Scope:** Custom connector plugins are deployed at the organization level and are accessible across clusters. * **ZIP format:** If uploading a `.zip` file, the archive must follow the Confluent-supported [file structure](#cc-byoc-archive-structure). Use the [Maven Packaging Plugin](https://docs.confluent.io/kafka-connectors/self-managed/confluent-hub/component-archive.html#maven-packaging-plugin) to automatically format and package your connector files into the required structure. For manual packaging rules, manifest file specifications, and sample structures, see the [Confluent Marketplace Component Archive Specification](https://docs.confluent.io/kafka-connectors/self-managed/confluent-hub/component-archive.html). ### Connector plugin archive structure A connector plugin archive contains the following files and directories: * `manifest.json`: Contains mandatory archive metadata and specifications. * `assets/`: Contains icons and logo assets for display in the UI. * `doc/`: Contains documentation, licensing information, and notice files. * `etc/`: Contains sample connector configuration properties. * `lib/`: Contains all Java dependencies and JAR files needed to run the connector. #### NOTE * The `lib/` directory must include all JAR dependencies required by the connector, including external client libraries, custom Single Message Transformations (SMTs), and non-Kafka JAR files. * Standard SMTs from the [Confluent Transforms archive](https://www.confluent.io/hub/confluentinc/connect-transforms) are pre-installed and do not need to be included in your package. The following examples show the archive structure and files. ![Connector archive structure](images/ccloud-byoc-hub-connector-archive-structure.png) ```bash confluentinc-kafka-connect-datagen-0.6.0/ assets/ confluent.png doc/ LICENSE README.md licenses licenses.html ... notices etc/ connector_campaign_finance.config connector_credit_cards.config connector_custom.config connector_inventory.config ... lib/ automaton-1.11-8.jar avro-1.11.0.jar avro-random-generator-0.4.1.jar checker-qual-3.5.0.jar common-utils-7.2.0.jar commons-compress-1.21.jar error_prone_annotations-2.3.4.jar ... swagger-annotations-2.1.10.jar manifest.json ``` #### manifest.json file The `manifest.json` file defines essential metadata for your custom connector plugin and can include many fields. The following table describes key manifest fields: | Field | Definition | Required/Optional | |---------------------|----------------------------------------------------------|---------------------| | `version` | Sets the plugin version string. | Required | | `documentation_url` | URL pointing to plugin documentation or repository. | Optional | | `kafka_version` | The Kafka version the connector plugin compiles against. | Optional | | `java_version` | The Java runtime version the connector plugin targets. | Optional | Example file: ```json { "name": "kafka-connect-my-custom-connector", "version": "1.0.0", "documentation_url": "https://github.com/myorg/kafka-connect-my-custom-connector/blob/main/README.md", "kafka_version": "3.6.0", "java_version": "17", ... } ``` ## Upload and launch a custom connector After you package your custom [connector plugin](#cc-byoc-packaging), upload the packaged `.zip` or `.jar` file to Confluent Cloud. When the upload finishes, users across your organization can configure and launch instances of the connector. ### Prerequisites Before you begin, ensure you have: * **Cluster access:** Access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/) cluster in a [supported region](custom-connector-fands.md#cc-byoc-regions). * **Permissions:** An administrator RBAC role with [permissions to upload custom connectors](../managed-connector-rbac.md#managed-connector-rbac-role-mappings). * **File format and structure:** Archives in `.zip` or `.jar` format not exceeding 250 MB. ZIP files must follow the required [file structure](#cc-byoc-packaging). You can upload a custom connector plugin archive from either the **Artifacts** page or the **Connector Plugins** page. ### Artifacts 1. In your Confluent Cloud environment, select **Artifacts**, and then select the **Custom connectors** tab. 2. Click **Upload Plugin Artifact**. ![Add connector plugin on Artifacts page](images/ccloud-artifacts.png) 3. On the **Upload custom connector plugin** page, select **Create new plugin**. 4. Enter a **Plugin name**. 5. Enter an optional **Description**. 6. Select the **Cloud provider**. 7. Click **Upload JAR file** and select your connector archive. Confluent Cloud scans the uploaded archive and extracts the connector classes and configurations. #### NOTE Archives must be in `.zip` or `.jar` format and cannot exceed 250 MB. 8. Click **Continue**. Confluent Cloud adds the connector plugin to the **Artifacts** page. To add a version to an existing plugin, see [Add a custom connector plugin version](custom-connector-manage.md#cc-byoc-add-plugin-version). ### Connector Plugins 1. Navigate to the **Connectors** page in your Confluent Cloud cluster. 2. Optional. If existing connectors are listed, click **Add Connector**. 3. On the **Connector Plugins** page, click **Add custom connector plugin**. ![Connector Plugins page](images/ccloud-connector-plugins.png) 4. Enter the required details: - Enter a **Plugin name**. - Enter an optional **Description**. - Select the **Cloud provider**. - Click **Upload JAR file** and select your connector archive. Confluent Cloud scans the uploaded archive and extracts the connector classes and configurations. #### NOTE Archives must be in `.zip` or `.jar` format and cannot exceed 250 MB. 5. Click **Continue**. Confluent Cloud adds the connector plugin to the **Connector Plugins** page. ## Configure and launch a connector ### Prerequisites Before configuring and launching a custom connector, ensure you have: * Access to a [Confluent Cloud](https://www.confluent.io/confluent-cloud/) cluster in a [supported region](custom-connector-fands.md#cc-byoc-regions). * Authentication credentials for the target data system. * Required egress [network endpoints](#cc-byoc-endpoints) configured for the target system. * [Schema Registry](../../get-started/schema-registry.md#cloud-sr-config) enabled on your cluster if using Schema Registry-based formats, such as Avro, JSON Schema, or Protobuf. For details, see [Schema Registry integration](custom-connector-fands.md#cc-byoc-sr-integration). * The list of required connector configuration properties. * The Confluent CLI installed and authenticated to your cluster. See [Install the Confluent CLI](https://docs.confluent.io/confluent-cli/current/install.html). #### NOTE Many connectors require target Kafka topics to exist before launch. Check your connector documentation for specific requirements. On the **Connector Plugins** page, filter plugins by **Custom**, and then select the plugin you want to configure, or hover over the connector card and click **Get Started**. ![Select the connector plugin to configure on Connector Plugins page](images/ccloud-select-custom-connector.png) The add connector configuration step is displayed. ![Add connector configuration step](images/ccloud-add-connector-details.png) ### Step 1: Add connector details In the **Connector details** step, select plugin details, such as version, type, Kafka, and JDK versions. ### Step 2: Add credentials In the **Kafka access** step, create or select an API key. You can choose one of the following options: - **My Account** to generate an API key and secret that your connector can use to communicate with your Kafka cluster. - **Service Account** to select or create a service account for your connector to communicate with your Kafka cluster. This option is recommended for production use. - **Use an existing API key** to enter an API key and secret pair you have stored. ### Step 3: Add configuration properties During the **Configuration** step, Confluent Cloud automatically populates the configuration properties extracted from your connector plugin. Enter the required values for your target deployment. The displayed properties depend on the selected plugin version and `connector.class`. #### Manage Schema Registry settings To manage schema serialization and deserialization formats, configure the following properties: * Key converter class * Value converter class * Schema Registry auto mode For more information, see [Schema Registry integration](custom-connector-fands.md#cc-byoc-sr-integration). #### Configure advanced properties Expand **Show advanced configurations** to configure the following: * **Optional connector properties:** Configure parameters defined with low or medium importance in the connector file that are not required for basic setup. * **Additional properties:** Add custom properties to override default connector behavior. For supported configuration overrides, see [Override configuration properties](custom-connector-manage.md#cc-bring-your-connector-override-configs). * **Single Message Transforms (SMTs):** Add transformation rules to change message keys, values, or headers. For details, see [Single Message Transformations](../single-message-transforms.md#cc-single-message-transforms-list). ### Step 4: Add networking endpoints In the **Networking** step, specify the connection endpoints required for the connector to reach your source or sink system. Endpoints must use a fully qualified domain name (FQDN). Protocol specification is optional and defaults to `TCP`. For details, see [Endpoint requirements](#endpoint-requirements). Use the following syntax pattern: ```text :[:] ``` **Examples:** * **Amazon S3:** `s3.us-west-2.amazonaws.com:443` * **Amazon RDS:** `mydatabase.abc123ecs2.us-west-2.rds.amazonaws.com:1433:TCP` * **Wildcard pattern:** `*database.abc123ecs2.us-west-2.rds.amazonaws.com:1433:TCP` #### NOTE Confluent automatically applies a single-level wildcard to the leftmost label of a domain. For example, specifying `s3.us-west-2.amazonaws.com:443` implicitly permits access to `*.s3.us-west-2.amazonaws.com:443`. Entering subdomains or multi-level wildcards, such as `*.s3.us-west-2.amazonaws.com:443`, as input is not supported. #### Endpoint requirements * Separate multiple endpoints using a semicolon. * Separate the hostname, ports, and protocols with colons. * Separate multiple ports using a comma. * Do not include `HTTP` or `HTTPS` with the endpoint. * The protocol is optional and defaults to `TCP`. Using `ANY` is not allowed. Note the following wildcard requirements: * Only one wildcard is allowed per endpoint. * The wildcard is only allowed in the leftmost element of the endpoint entry. * Using a wildcard for a subdomain is not allowed. The endpoint is checked for valid characters. Hostnames are checked for valid patterns (excluding a wildcard) and against the [Web Risk API](https://cloud.google.com/web-risk/docs/reference/rest). For more information, see [Web Risk](https://cloud.google.com/web-risk). #### IMPORTANT Confluent routinely scans uploaded custom connectors to detect security risks and interactions with malicious endpoints. If malicious activity is detected, Confluent reserves the right to delete the connector immediately. ### Step 5: Configure sizing In the **Sizing** step, enter the number of **Tasks** to allocate for the connector. * **Minimum tasks:** Enter `1`. This is recommended for testing or low-volume workloads. * **Scaling for throughput:** Increase the task count to process data in parallel across multiple partitions or files. #### NOTE Connector memory of 2 GB is shared across all allocated tasks. Increasing the number of tasks increases throughput and hourly costs, but splits the shared memory among active tasks. ### Step 6: Review and launch In the **Review and launch** step, review the provisioned details and select **Continue**. Connector configurations marked with an asterisk (\*) cannot be changed after you launch your connector. The connector’s status changes to running after provisioning completes. ![Connector card showing Provisioning status](images/ccloud-byoc-connector-provisioning.png) #### NOTE If the connector remains in the **Provisioning** state for an extended period, verify that all required properties are specified correctly without typos or syntax errors. For details on updating configuration settings, see [Modify a custom connector configuration](custom-connector-manage.md#cc-byoc-modify-config). ### Next steps * To search, monitor, reconfigure, or delete a connector or plugin after launch, see [Manage Custom Connectors for Confluent Cloud](custom-connector-manage.md#cc-bring-your-connector-manage). * To automate custom connector or plugin tasks from the command line, see [Confluent Cloud API and Confluent CLI for Custom Connectors](custom-connector-cli.md#cc-bring-your-connector-cli). * For custom connector and custom connector plugin API information and examples, see the [Confluent Cloud API for Connect Usage Examples](../connect-api-section.md#ccloud-connect-api).