Manage Custom Connectors for Confluent Cloud

To manage running connectors, refer to the following sections.

Search for a custom connector

To find a custom connector, first show all connectors, and then select Cloud custom in the Filter by: Deployment dropdown list.

Filter by deployment

Get notifications

Confluent Cloud provides out-of-the-box notification capability. You can be notified when a connector status changes to RUNNING or FAILED. For more information, see Notifications for Confluent Cloud.

Modify a custom connector configuration

Complete the following steps to modify a custom connector configuration.

1. Open the connector Overview page.

   

   Connector overview

2. Go to Settings. You can update the number of tasks, change networking endpoints, update credentials, and change or add the connector configuration properties.

3. To update the custom connector configuration, click the edit icon under Custom configuration.

   

   Edit custom configuration

4. Modify a property or click Add key/value pair and add a new property. You can also modify the JSON configuration directly on the JSON tab.

   

   Update Key/value pairs

5. Click Save changes and then click Apply changes. The connector restarts using the updated configuration.

Override configuration properties

By default, connect-level Kafka clients inherit their client configurations from the worker properties, but customers can optionally provide producer.override., consumer.override., and admin.override. connector properties to override worker-level settings. The following tabs list the configuration overrides you can use with custom connectors.

AdminConsumer (sink)Producer (source)

• connections.max.idle.ms
• default.api.timeout.ms
• receive.buffer.bytes
• reconnect.backoff.max.ms
• reconnect.backoff.ms
• request.timeout.ms
• retries
• retry.backoff.ms
• send.buffer.bytes

For example, "admin.override.retry.backoff.ms": "300". For property definitions, see Admin Configuration Properties.

Update networking endpoints

Complete the following steps to update egress endpoints. You can use the fully-qualified domain name (FQDN) or IP address when updating networking endpoints.

1. Open the connector Overview page.

   

   Connector overview

2. Go to Settings. You can update the number of tasks, change networking endpoints, update credentials, and change or add the connector configuration properties.

3. To update the custom connector configuration, click the edit icon under Networking.

   

   Edit egress endpoints

4. Update an endpoint or click Add connection endpoint and add a new endpoint. For details about egress endpoints, see endpoints.

   

   Update endpoints

5. Click Save changes and then click Apply changes. The connector restarts using the updated endpoints.

View the app log topic

When you launch a custom connector, Confluent Cloud creates a log topic file with the name clcc-<cluster-ID>-app-logs. The log topic captures the following log messages:

• Connector code log messages
• Log messages emitted by code the connector calls
• Connect worker process log messages

Important

Customers are responsible for all charges related to using the app log topic with a custom connector. For billing details, see Custom connector cluster.

View from logs tab

Complete the following steps to view log messages:

1. Open the connector Overview page and click Logs.

   

   All log messages are displayed.

2. Change the view if you only want to see important messages. The messages timestamp in the log can be local time or UTC. If you want to view the full stack trace for an error message, click Show stacktrace.

   

   Custom connector logs

3. The full stack trace message is displayed.

   

   View Custom Connector stack trace

4. Click Export to download log messages (JSON and CSV files).

View in app log topic

Complete the following steps to view messages in the log topic.

Important

Customers are responsible for all charges related to using the app log topic with a custom connector. For billing details, see Custom connector cluster.

1. Open the Topics page and click the log topic name.

   

   Select log topic

2. Click Messages.

   

   View log topic messages

3. Expand the message you want to view.

   

   View message details

The examples show Connect worker process log messages.

Export log messages

The connector stores log messages in a Kafka topic. You can export log data using any of the following options:

• Export logs using a Confluent connector: For example the Elasticsearch Service Sink connector for Confluent Cloud, or the Elasticsearch Service Sink connector for Confluent Platform can export logs to Elasticsearch. Several additional connectors are available that may also be used for exporting logs.
• Create a custom integration using the Kafka REST API for topics to get log messages to a destination logs service.

To manually configure a destination service to capture logs, you will need the following:

• Bootstrap server endpoint: This is provided on the Cluster Settings page. For example, pkc-abc123.<aws-region>.aws.confluent.cloud:9092. You can also get this information using the following Confluent CLI command:

  confluent kafka cluster describe
  

• Log topic name: Get this from the topics page. For example, clcc-<cluster-ID>-app-logs. You can also get this information using the following Confluent CLI command:

  confluent kafka topic list
  

This information is also provided in the UI in Cluster settings.

View cluster settings

View metrics

Basic message and production throughput metrics are shown on the connector overview page.

Connect worker metrics

Click Metrics to view additional metrics for the Kafka Connect worker. Worker health metrics are available for up to seven days.

Tip

For Confluent Cloud Metrics API examples, see the Example Queries.

View throughput metrics

Scroll for other metrics.

View other metrics

Custom connector metrics

You can view metrics for a specific connector and export metrics.

1. Click Explore metrics on the connector overview page.

   

   Explore metrics

2. Under Metric, select a custom connector metric you want to view.

   

   View metric

3. Under Resource, search for and select a custom connector. The selected metric for the connector is displayed. Custom connector metrics are available for up to seven days.

   

   Select custom connector

You can export metrics for your custom connector. Click New integration and choose the monitoring platform you want to use.

Delete a custom connector

Complete the following steps to delete the custom connector instance. If you want to delete a custom connector plugin, see Delete a custom connector plugin.

Note

Deleting a connector does not delete the associated Kafka app log topic.

Open the connector Overview page. Click Settings. The option to delete a connector is located at the bottom of the Settings screen.

1. Click Delete connector.

   

   Delete connector on Settings

2. Enter the connector name on the confirmation screen and click Confirm.

View a custom connector plugin ID

1. Go to the main Connectors page.

2. Select the plugin and click Edit plugin.

   

   Click Edit plugin

   The plugin ID and other details are displayed.

   

   Show plugin details

Delete a custom connector plugin

1. Go to the main Connectors page.

   

   Delete connector plugin

2. Select the plugin and click Edit plugin.

3. Click Delete plugin.