# Manage Custom Connectors for Confluent Cloud To manage running connectors, refer to the following sections. ## Search for a custom connector To find a custom connector, first show all connectors, and then select **Cloud custom** in the **Filter by: Deployment** dropdown list. ![Filter by deployment](images/ccloud-byoc-add-connector-plugin-filter-deployment.png) ## Get notifications Confluent Cloud provides out-of-the-box notification capability. You can be notified when a connector status changes to RUNNING or FAILED. For more information, see [Notifications for Confluent Cloud](../../monitoring/configure-notifications.md#ccloud-notifications). ## Modify a custom connector configuration Complete the following steps to modify a custom connector configuration. 1. Open the connector **Overview** page. ![Connector overview](images/ccloud-byoc-modify-config-overview-settings.png) 2. Go to **Settings**. You can update the number of tasks, change networking endpoints, update credentials, and change or add the connector configuration properties. 3. To update the custom connector configuration, click the edit icon under **Custom configuration**. ![Edit custom configuration](images/ccloud-byoc-modify-config-edit-icon.png) 4. Modify a property or click **Add key/value pair** and add a new property. You can also modify the JSON configuration directly on the **JSON** tab. ![Update Key/value pairs](images/ccloud-byoc-modify-config-edit-configuration.png) 5. Click **Save changes** and then click **Apply changes**. The connector restarts using the updated configuration. ## Override configuration properties By default, connect-level Kafka clients inherit their client configurations from the worker properties, but customers can optionally provide `producer.override.`, `consumer.override.`, and `admin.override.` connector properties to override worker-level settings. The following tabs list the configuration overrides you can use with custom connectors. ### Admin - connections.max.idle.ms - default.api.timeout.ms - receive.buffer.bytes - reconnect.backoff.max.ms - reconnect.backoff.ms - request.timeout.ms - retries - retry.backoff.ms - send.buffer.bytes For example, `"admin.override.retry.backoff.ms": "300"`. For property definitions, see [Admin Configuration Properties](/platform/current/installation/configuration/admin-configs.html). ### Consumer (sink) - auto.offset.reset - connections.max.idle.ms - default.api.timeout.ms - fetch.max.bytes - fetch.max.wait.ms - fetch.min.bytes - heartbeat.interval.ms - max.partition.fetch.bytes - max.poll.interval.ms - max.poll.records - receive.buffer.bytes - reconnect.backoff.max.ms - reconnect.backoff.ms - request.timeout.ms - retry.backoff.ms - session.timeout.ms For example, `"consumer.override.auto.offset.reset": "latest"`. For property definitions, see [Consumer Configuration Properties](/platform/current/installation/configuration/consumer-configs.html). ### Producer (source) - acks - batch.size - buffer.memory - compression.type - connections.max.idle.ms - delivery.timeout.ms - enable.idempotence - linger.ms - max.block.ms - max.in.flight.requests.per.connection - max.request.size - receive.buffer.bytes - reconnect.backoff.max.ms - reconnect.backoff.ms - request.timeout.ms - retries - retry.backoff.ms For example, `"producer.override.max.request.size": "5000"`. For property definitions, see [Producer Configuration Properties](/platform/current/installation/configuration/producer-configs.html). ## Update networking endpoints Complete the following steps to update egress endpoints. You can use the fully-qualified domain name (FQDN) or IP address when updating networking endpoints. 1. Open the connector **Overview** page. ![Connector overview](images/ccloud-byoc-modify-config-overview-settings.png) 2. Go to **Settings**. You can update the number of tasks, change networking endpoints, update credentials, and change or add the connector configuration properties. 3. To update the custom connector configuration, click the edit icon under **Networking**. ![Edit egress endpoints](images/ccloud-byoc-modify-config-edit-icon-networking.png) 4. Update an endpoint or click **Add connection endpoint** and add a new endpoint. For details about egress endpoints, see [endpoints](custom-connector-qs.md#cc-byoc-endpoints). ![Update endpoints](images/ccloud-byoc-modify-config-edit-networking.png) 5. Click **Save changes** and then click **Apply changes**. The connector restarts using the updated endpoints. ## Custom connector logs You can view Confluent Cloud custom connector events directly through the Confluent Cloud Console or Confluent CLI. ### Using Confluent Cloud Console The logs provide a centralized view for monitoring the operational health and activity of your Confluent Cloud custom connectors. You can see the customer connector logs from: 1. Logs tab 2. App log topic #### View from logs tab To view log messages, open the connector **Overview** page and click **Logs**. ![Logs](images/ccloud-byoc-log.png) The messages timestamp in the log can be in local time or UTC. If you want to view the full stack trace for an error message, expand **View stack trace**. ![Error Logs with stack trace](images/ccloud-byoc-log-stacktrace-1.png) If you want to download the log messages, click **JSON** or **CSV** to export the file in the respective formats. If you want to navigate to the logs topic of the connector, click **Logs Topic**. ![Select log topic from Logs tab](images/ccloud-byoc-logs-topic.gif) #### View in app log topic When you launch a custom connector, Confluent Cloud creates a log topic file with the name `clcc--app-logs`. The log topic captures the following log messages: - Connector code log messages. - Log messages emitted by code the connector calls. - Connect worker process log messages. #### IMPORTANT Customers are responsible for all charges related to using the app log topic with a custom connector. For billing details, see [Custom connector cluster](../../billing/billing-dimensions.md#byoc-billing). To view messages in the log topic, open the **Topics** page and click the log topic name. ![Select log topic](images/ccloud-byoc-log-topic-selection.png) Click **Messages** to view log topic messages. ![View log topic messages](images/ccloud-byoc-log-topic-messages.png) You can also click the message timestamp to view more log details. ![View message details](images/ccloud-byoc-log-topic-message-detail.png) ### Using Confluent CLI The Confluent CLI allows you to view connector logs with various search criteria, similar to the Confluent Cloud Console. Additionally, it allows you to store logs in an output file. For more information, see [Connect Logs CLI](https://docs.confluent.io/confluent-cli/current/command-reference/connect/confluent_connect_logs.html). #### Limitations - A rate limit of 5 requests/minute applies per connector. - Log data can be queried only for the past 3 days. #### Query examples The following examples show how to view connector log events using the [Confluent Cloud CLI](https://docs.confluent.io/confluent-cli/current/overview.html). The example assumes the following: - You have the Confluent CLI installed and configured for your environment. - You have the Connector ID (for example, `lcc-123456`) for the connector whose logs you want to view. 1. **Query logs by time range**: Allows you to query connector logs with a specific log level within a defined time window. ```bash confluent connect logs --level --start-time "" --end-time "" ``` For example, to query for `ERROR` level logs for connector `lcc-123456` between February 1, 2025, 00:00:00Z and February 1, 2025, 23:59:59Z: ```bash confluent connect logs lcc-123456 --level ERROR --start-time "2025-02-01T00:00:00Z" --end-time "2025-02-01T23:59:59Z" ``` Example output: ```bash Found 2 log entries for connector lcc-123456: Timestamp | Level | Task ID | Message | Exception ---------------------------+--------+---------+-----------------------------------+------------------------------------------------------ 2025-02-01T00:00:00.757Z | ERROR | task-0 | WorkerSourceTask{id=lcc-123456-0} | org.apache.kafka.connect.errors.ConnectException: ... | | | Task threw an uncaught and | | | | unrecoverable exception. Task is | | | | being killed and will not recover | | | | until manually restarted | 2025-02-01T00:00:00.761Z | ERROR | task-0 | WorkerSourceTask{id=lcc-123456-0} | org.apache.kafka.connect.errors.ConnectException: ... | | | Task threw an uncaught and | | | | unrecoverable exception. Task is | | | | being killed and will not recover | | | | until manually restarted | ``` 2. **Filter by multiple log levels**: Lets you query connector logs by specifying multiple log levels using the | (OR) operator within the defined time window. ```bash confluent connect logs --level "|" --start-time "" --end-time "" ``` For example, to query for `ERROR` or `INFO` level logs for connector `lcc-123456` between February 1, 2025, 00:00:00Z and February 1, 2025, 23:59:59Z: ```bash confluent connect logs lcc-123456 --level "ERROR|INFO" --start-time "2025-02-01T00:00:00Z" --end-time "2025-02-01T23:59:59Z" ``` Example output: ```bash Found 2 log entries for connector lcc-123456: Timestamp | Level | Task ID | Message | Exception ---------------------------+--------+---------+-----------------------------------+------------------------------------------------------ 2025-02-01T00:00:00.757Z | INFO | task-0 | WorkerSourceTask{id=lcc-123456-0} | | | | Committing offsets for 130 | | | | acknowledged messages | 2025-02-01T00:00:00.761Z | ERROR | task-0 | WorkerSourceTask{id=lcc-123456-0} | org.apache.kafka.connect.errors.ConnectException: ... | | | Task threw an uncaught and | | | | unrecoverable exception. Task is | | | | being killed and will not recover | | | | until manually restarted | ``` 3. **Search for specific text and save to a file**: Allows you to query connector logs with a specific log level and search for particular text within the logs. You can also save the results to an output file. You can use regular expressions for the search text. ```bash confluent connect logs --level --search-text "" --start-time "" --end-time "" --output-file ``` For example, to query for `ERROR` level logs for connector `lcc-123456` containing “example error” between February 1, 2025, 00:00:00Z and February 1, 2025, 23:59:59Z, and save them to `errors.json`: ```bash confluent connect logs lcc-123456 --level "ERROR" --search-text "example error" --start-time "2025-02-01T00:00:00Z" --end-time "2025-02-01T23:59:59Z" --output-file errors.json ``` Example output: ```bash Appended 1 log entries to file: errors.json $ cat errors.json { "Timestamp":"2025-02-01T00:00:00.757Z", "Level":"ERROR","TaskId":"task-0", "Message":"example error 1", "Exception":"" } ``` Example with a regular expression `"exa*"`: ```bash confluent connect logs lcc-123456 --level "ERROR" --search-text "exa*" --start-time "2025-02-01T00:00:00Z" --end-time "2025-02-01T23:59:59Z" --output-file errors.json ``` Example output: ```bash Appended 1 log entries to file: errors.json $ cat errors.json { "Timestamp":"2025-02-01T00:00:00.757Z", "Level":"ERROR","TaskId":"task-0", "Message":"example error 1", "Exception":"" } ``` 4. **Navigate through log pages**: Allows you to query subsequent pages of connector logs for the same query by using the `--next` flag. Continue executing the command with this flag until `No logs found for the current query` is displayed. ```bash confluent connect logs --level --start-time "" --end-time "" --next ``` For example, to navigate to the next page of `ERROR` level logs for connector `lcc-123456` between February 1, 2025, 00:00:00Z and February 1, 2025, 23:59:59Z: ```bash confluent connect logs lcc-123456 --level ERROR --start-time "2025-02-01T00:00:00Z" --end-time "2025-02-01T23:59:59Z" --next ``` ```bash Found 2 log entries for connector lcc-123456: Timestamp | Level | Task ID | Message | Exception ---------------------------+--------+---------+-----------------------------------+------------------------------------------------------ 2025-02-01T23:59:59.850Z | ERROR | task-0 | WorkerSourceTask{id=lcc-123456-0} | org.apache.kafka.connect.errors.ConnectException: ... | | | Task threw an uncaught and | | | | unrecoverable exception. Task is | | | | being killed and will not recover | | | | until manually restarted | 2025-02-01T23:59:59.860Z | ERROR | task-0 | WorkerSourceTask{id=lcc-123456-0} | org.apache.kafka.connect.errors.ConnectException: ... | | | Task threw an uncaught and | | | | unrecoverable exception. Task is | | | | being killed and will not recover | | | | until manually restarted | ``` ### Export log messages The connector stores log messages in a Kafka topic. You can export log data using any of the following options: * Export logs using a Confluent connector: For example the [Elasticsearch Service Sink connector for Confluent Cloud](../cc-elasticsearch-service-sink.md#cc-elasticsearch-service-sink), or the [Elasticsearch Service Sink connector for Confluent Platform](https://docs.confluent.io/kafka-connectors/elasticsearch/current/overview.html) can export logs to Elasticsearch. Several additional connectors are available that may also be used for exporting logs. * Create a custom integration using the [Kafka REST API for topics](https://docs.confluent.io/cloud/current/ccloud/topic-v-3/) to get log messages to a destination logs service. To manually configure a destination service to capture logs, you will need the following: * Bootstrap server endpoint: This is provided on the **Cluster Settings** page. For example, `pkc-abc123..aws.confluent.cloud:9092`. You can also get this information using the following Confluent CLI command: ```bash confluent kafka cluster describe ``` * Log topic name: Get this from the topics page. For example, `clcc--app-logs`. You can also get this information using the following Confluent CLI command: ```bash confluent kafka topic list ``` This information is also provided in the UI in **Cluster settings**. ![View cluster settings](images/ccloud-byoc-log-cluster-settings.png) ## View metrics Basic message and production throughput metrics are shown on the connector overview page. ### Connect worker metrics Click **Metrics** to view additional metrics for the Kafka Connect worker. Worker health metrics are available for up to seven days. ![View throughput metrics](images/ccloud-byoc-metrics-overview.png) Scroll for other metrics. ![View other metrics](images/ccloud-byoc-metrics-selection.png) ### Custom connector metrics You can view metrics for a specific connector and export metrics. 1. Click **Explore metrics** on the connector overview page. ![Explore metrics](images/ccloud-byoc-metrics-explore-button.png) 2. Under **Metric**, select a custom connector metric you want to view. ![View metric](images/ccloud-byoc-metrics-select-metric.png) 3. Under **Resource**, search for and select a custom connector. The selected metric for the connector is displayed. Custom connector metrics are available for up to seven days. ![Select custom connector](images/ccloud-byoc-metrics-select-connector.png) You can export metrics for your custom connector. Click **New integration** and choose the monitoring platform you want to use. ## Delete a custom connector Complete the following steps to delete the custom connector instance. If you want to delete a custom connector plugin, see [Delete a custom connector plugin](#cc-byoc-delete-plugin). #### NOTE Deleting a connector does not delete the associated Kafka [app log topic](#cc-byoc-view-the-log). Open the connector **Overview** page. Click **Settings**. The option to delete a connector is located at the bottom of the Settings screen. 1. Click **Delete connector**. ![Delete connector on Settings](images/ccloud-byoc-delete-connector-alt.png) 2. Enter the connector name on the confirmation screen and click **Confirm**. ## View a custom connector plugin ID 1. Go to the main **Connectors** page. 2. Select the plugin and click **Edit plugin**. ![Click Edit plugin](images/ccloud-byoc-delete-plugin.png) The plugin ID and other details are displayed. ![Show custom connector plugin details](images/ccloud-byoc-find-plugin-id.png) ## Delete a custom connector plugin 1. Go to the main **Connectors** page. ![Delete connector plugin](images/ccloud-byoc-delete-plugin.png) 2. Select the plugin and click **Edit plugin**. 3. Click **Delete plugin**.