Manage Custom Connectors for Confluent Cloud

To manage running connectors, refer to the following sections.

Search for a custom connector

To find a custom connector, first show all connectors, and then select Cloud custom in the Filter by: Deployment dropdown list.

Filter by deployment

Get notifications

Confluent Cloud provides out-of-the-box notification capability. You can be notified when a connector status changes to RUNNING or FAILED. For more information, see Notifications for Confluent Cloud.

Modify a custom connector configuration

Complete the following steps to modify a custom connector configuration.

1. Open the connector Overview page.

   

   Connector overview

2. Go to Settings. You can update the number of tasks, change networking endpoints, update credentials, and change or add the connector configuration properties.

3. To update the custom connector configuration, click the edit icon under Custom configuration.

   

   Edit custom configuration

4. Modify a property or click Add key/value pair and add a new property. You can also modify the JSON configuration directly on the JSON tab.

   

   Update Key/value pairs

5. Click Save changes and then click Apply changes. The connector restarts using the updated configuration.

Update networking endpoints

Complete the following steps to update egress endpoints.

1. Open the connector Overview page.

   

   Connector overview

2. Go to Settings. You can update the number of tasks, change networking endpoints, update credentials, and change or add the connector configuration properties.

3. To update the custom connector configuration, click the edit icon under Networking.

   

   Edit egress endpoints

4. Update an endpoint or click Add connection endpoint and add a new endpoint. For details about egress endpoints, see endpoints.

   

   Update endpoints

5. Click Save changes and then click Apply changes. The connector restarts using the updated endpoints.

View the app log topic

When you launch a custom connector, Confluent Cloud creates a log topic file with the name clcc-<cluster-ID>-app-logs. The log topic captures the following log messages:

• Connector code log messages
• Log messages emitted by code the connector calls
• Connect worker process log messages

Important

Customers are responsible for all charges related to using the app log topic with a custom connector. For billing details, see Custom connector cluster.

View from logs tab

Complete the following steps to view log messages:

1. Open the connector Overview page and click Logs.

   

   Connector overview

   All log messages are displayed.

2. Change the view if you only want to see important messages. The messages timestamp in the log can be local time or UTC.

   

   Custom connector logs

3. Click Export to download log messages (JSON and CSV files).

View in app log topic

Complete the following steps to view messages in the log topic.

Important

Customers are responsible for all charges related to using the app log topic with a custom connector. For billing details, see Custom connector cluster.

1. Open the Topics page and click the log topic name.

   

   Select log topic

2. Click Messages.

   

   View log topic messages

3. Expand the message you want to view.

   

   View message details

The examples show Connect worker process log messages.

Export log messages

The connector stores log messages in a Kafka topic. You can export log data using any of the following options:

• Export logs using a Confluent connector: For example the Elasticsearch Service Sink connector for Confluent Cloud, or the Elasticsearch Service Sink connector for Confluent Platform can export logs to Elasticsearch. Several additional connectors are available that may also be used for exporting logs.
• Create a custom integration using the Kafka REST API for topics to get log messages to a destination logs service.

To manually configure a destination service to capture logs, you will need the following:

• Bootstrap server endpoint: This is provided on the Cluster Settings page. For example, pkc-abc123.<aws-region>.aws.confluent.cloud:9092. You can also get this information using the following Confluent CLI command:

  confluent kafka cluster describe
  

• Log topic name: Get this from the topics page. For example, clcc-<cluster-ID>-app-logs. You can also get this information using the following Confluent CLI command:

  confluent kafka topic list
  

This information is also provided in the UI in Cluster settings.

View cluster settings

View metrics

Basic message and production throughput metrics are shown on the connector overview page. Click Metrics to view additional metrics for the Kafka Connect worker.

Tip

For Confluent Cloud Metrics API examples, see the Example Queries.

View throughput metrics

Scroll for other metrics.

View other metrics

You can view metrics for up to the last seven days.

You can export metrics using integrations with the metrics API.

1. Click Explore metrics.

   

   Explore metrics

2. Under Explore available metrics, search for and select a custom connector.

   

   Select custom connector

3. Select the metric you want to view. The example shows Connector received bytes.

   

   View metric

Delete a custom connector

Complete the following steps to delete the custom connector instance. If you want to delete a custom connector plugin, see Delete a custom connector plugin.

Note

Deleting a connector does not delete the associated Kafka app log topic.

Open the connector Overview page. Click Settings. The option to delete a connector is located at the bottom of the Settings screen.

1. Click Delete connector.

   

   Delete connector on Settings

2. Enter the connector name on the confirmation screen and click Confirm.

Delete a custom connector plugin

1. Go to the main Connectors page.

   

   Delete connector plugin

2. Select the plugin and click Edit plugin.

3. Click Delete plugin.