Manage Custom Connectors for Confluent Cloud

Managing a custom connector in Confluent Cloud covers two related tasks:

Manage connector instances: Update running configurations, reconfigure networking, review system metrics, and delete retired instances.
Manage connector plugins: Manage uploaded JAR or ZIP archives, update metadata, and publish new plugin versions.

Manage connector instances

After you launch a custom connector, use the Confluent Cloud Console to:

Locate and inspect active custom connector instances.
Modify settings, including runtime configurations and networking rules.
Monitor performance using integrated logs and system metrics.
Decommission or delete connectors when they are no longer needed.

Search for a custom connector

Navigate to the Connectors page in your Confluent Cloud cluster.
If existing connectors are listed, click Add Connector.
On the Connector Plugins page, from the Deployment search filter dropdown, select Custom to list the custom connectors in your cluster.

Deployment filter dropdown with Custom option selected — Filter by deployment

Modify a custom connector configuration

Use the following steps to update tasks, networking, credentials, or configuration properties for a running custom connector.

On your cluster page in the Cloud Console, select Connectors.
Select your custom connector.
On the connector Overview page, select Settings.

Connector overview
Under Configuration, click the edit icon.
Modify the property values.
Click Save changes to save the custom configurations and then click Apply changes on the Settings page. The connector restarts using the updated configuration.

Override configuration properties

By default, connect-level Apache Kafka® clients inherit their client configurations from the worker properties. You can optionally override worker-level settings using producer.override., consumer.override., and admin.override. connector properties. The following tabs list the configuration overrides you can use with custom connectors.

Admin

connections.max.idle.ms
default.api.timeout.ms
receive.buffer.bytes
reconnect.backoff.max.ms
reconnect.backoff.ms
request.timeout.ms
retries
retry.backoff.ms
send.buffer.bytes

For example, "admin.override.retry.backoff.ms": "300". For property definitions, see Admin Configuration Properties.

Consumer (sink)

auto.offset.reset
connections.max.idle.ms
default.api.timeout.ms
fetch.max.bytes
fetch.max.wait.ms
fetch.min.bytes
heartbeat.interval.ms
max.partition.fetch.bytes
max.poll.interval.ms
max.poll.records
receive.buffer.bytes
reconnect.backoff.max.ms
reconnect.backoff.ms
request.timeout.ms
retry.backoff.ms
session.timeout.ms

For example, "consumer.override.auto.offset.reset": "latest". For property definitions, see Consumer Configuration Properties.

Producer (source)

acks
batch.size
buffer.memory
compression.type
connections.max.idle.ms
delivery.timeout.ms
enable.idempotence
linger.ms
max.block.ms
max.in.flight.requests.per.connection
max.request.size
receive.buffer.bytes
reconnect.backoff.max.ms
reconnect.backoff.ms
request.timeout.ms
retries
retry.backoff.ms

For example, "producer.override.max.request.size": "5000". For property definitions, see Producer Configuration Properties.

Update networking endpoints

You can use the fully qualified domain name (FQDN) or IP address when updating networking endpoints. Complete the following steps to update egress endpoints.

On your cluster page in the Cloud Console, click Connectors and select your custom connector.
On the connector Overview page, select Settings.
Under Networking, click the edit icon.
Update an endpoint or click Add connection endpoint and add a new endpoint. For details about egress endpoints, see endpoints.

Update endpoints
Click Save changes to save the updated endpoint and then click Apply changes on the Settings page. The connector restarts using the updated endpoints.

Get notifications

Confluent Cloud provides out-of-the-box notification capability. You can receive notifications when connector status changes to RUNNING or FAILED. For more information, see Notifications for Confluent Cloud.

Custom connector logs

Logs give you a centralized view for monitoring the operational health and activity of your Confluent Cloud custom connectors. You can view Confluent Cloud custom connector events directly through the Confluent Cloud Console or Confluent CLI.

View custom connector logs using Confluent Cloud Console

In the Confluent Cloud Console, you can access the custom connector logs from the following pages:

Logs tab

To view log messages, open the connector Overview page and click Logs.

Log message list showing timestamp, level, and message columns — The **Logs** tab displays all log messages. Log timestamps appear in local time or UTC.

To view the full stack trace for an error log, expand View stack trace.
To download the log messages, click JSON or CSV to export the file in the respective formats.
To navigate to the logs topic of the connector, click Logs Topic.

App log topic

When you launch a custom connector, Confluent Cloud creates a log topic file with the name clcc-<connector-ID>-app-logs. The log topic captures the following log messages:

Connector code log messages.
Log messages emitted by code the connector calls.
Connect worker process log messages.

To view messages in the log topic:

In the left navigation menu of the Cloud Console, click Topics, and then select the log topic name on the Topics page.
Click Messages to view log topic messages.

Expanded detail view of a single log message — View message details to the highlighted step

You can also click the message timestamp to view more log details.

Important

You are responsible for all charges related to using the app log topic with a custom connector. For billing details, see Custom connector cluster.

View custom connector logs using Confluent CLI

Use the Confluent CLI to view connector logs with the same search criteria available in the Confluent Cloud Console, including filtering by time range or log level, searching log text, and saving results to a file. For prerequisites, limitations, and full command examples, see View custom connector logs in the API and CLI guide.

Export log messages

The connector stores log messages in a Kafka topic. You can export log data using any of the following options:

Export logs using a Confluent connector: For example, the Elasticsearch Service Sink connector for Confluent Cloud, or the Elasticsearch Service Sink connector for Confluent Platform can export logs to Elasticsearch. Several additional connectors are available that may also be used for exporting logs.
Create a custom integration using the Kafka REST API for topics to get log messages to a destination logs service.

To manually configure a destination service to capture logs, you need the following:

Bootstrap server endpoint: This is provided on the Cluster Settings page. For example, pkc-abc123.<aws-region>.aws.confluent.cloud:9092. You can also get this information using the following Confluent CLI command:
```
confluent kafka cluster describe
```
Log topic name: Get this from the Topics page. For example, clcc-<cluster-ID>-app-logs. You can also get this information using the following Confluent CLI command:
```
confluent kafka topic list
```

The bootstrap server endpoint and log topic name are also provided in the cluster overview page.

View metrics

The connector Overview page shows basic performance metrics.

To view additional metrics, select the Monitor tab. These metrics are available for up to seven days.

Tip

For Confluent Cloud Metrics API examples, see the Example Queries.

Delete a custom connector

If you want to delete a custom connector plugin instead of a connector instance, see Delete a custom connector plugin. Complete the following steps to delete the custom connector instance.

Note

Deleting a connector does not delete the associated Kafka app log topic.

Open the connector Overview page.
Click Settings.
Click Delete connector.
In the confirmation dialog, enter the connector name and click Confirm.

Manage connector plugins

Connector plugins are the uploaded JAR or ZIP archives used to create connector instances. In this section, you can view plugin details, edit a plugin’s name and description, add and manage plugin versions, and delete a plugin.

View the plugin details page

On the Environments page in the Cloud Console, click Artifacts.
Select the Custom connectors tab.

Show Custom Connectors tab
Select the plugin. The details page displays information, such as the plugin type, active connectors using the plugin, and supported cloud providers.

Edit a custom connector plugin name and description

Open the plugin details page. For details, see View the plugin details page.
Click Edit to update the plugin name and description.

Add a custom connector plugin version

Open the plugin details page. For details, see View the plugin details page.
Click Add Version.
Enter the version details and upload a JAR or ZIP file.
After the upload completes, accept the Confluent Cloud disclaimer, and then click Add Version. Confluent Cloud adds the new plugin version to the plugin page.

Manage custom connector plugin versions

Open the plugin details page. For details, see View the plugin details page.
Select the Versions tab to view the existing plugin versions.
Select a plugin version to view version details, such as version, type, documentation link, and connector class.

Delete a custom connector plugin

Note

You can delete a custom connector plugin only if it is not in use by any active connectors, or if it is in a FAILED state.

Open the plugin details page. For details, see View the plugin details page.
Select More actions and click Delete.
In the Delete Plugin pop-up window, enter the name of the plugin to confirm deletion, and then click Delete. This deletes the plugin from the Cloud Console.