View Confluent Cloud Connector Events

You can view the Confluent Cloud connector events in the Confluent Cloud Console. You can also consume events from a topic using the Confluent CLI, Java, or C/C++ for output to a display or other application. This feature is available for Basic, Standard, Enterprise and Dedicated Apache Kafka® clusters in Confluent Cloud.

Important

  • Viewing connector events is restricted to the OrganizationAdmin RBAC role. Viewing events is not available for other roles.
  • Connectors that fail during provisioning are not reported to the events topic.
  • Confluent will make non-breaking changes to the schema without advance notice. Breaking changes will be widely communicated at least 180 days in advance, and we will continue to maintain compatibility during this time. For additional details, see the event schema compatibility policy.

Events logged in the Connect events topic follow the cloudevents schema. The types of events logged in the topic are currently limited to io.confluent.logevents.connect.TaskFailed and io.confluent.logevents.connect.ConnectorFailed.

There are no ordering guarantees for events. Each record has a timestamp, but the events themselves may appear out of order when viewed. You can sort these based on the timestamp if needed. Note that you cannot sort the events directly in the Cloud Console.

For connector failure events, Connect attempts to resolve the failure automatically. For this reason, you may see an event get repeated if the issue cannot be resolved by Connect.

Note that connector events and audit logging operate from the Audit Log cluster. Consumption charges for both connector events and audit logging are combined. For more information, see Connector events and audit log.

In addition to what is provided in this document, the Cloud Console provides in-product documentation that steps through how to set this up. To view the in-product documentation for event logs, go to the Administration menu and select Connect log events.

Connector Events Admin Menu

View events in the Confluent Cloud Console

Each event is displayed in the Cloud Console as a JSON formatted record. Use the steps below to view an event for a Confluent Cloud connector:

  1. Select a connector in the Cloud Console to open the connector overview page.

  2. Click on Logs.

  3. Click and expand the event you want to view.

    Confluent Cloud Connector Event Log

    Connector Event Log

    The example error above indicates that the connector is not authorized to access AWS. You can use this message to correct the issue yourself or provide it to Confluent support for additional investigation.

    Most events are informational only and do not require corrective action.

  4. When you expand an event, the event stream is paused. Click on the play button to resume event streaming.

    Confluent Cloud Connector Event Play Button

    Pause/Play Button

Consume events from the event topic

You can consume events from the Connect events Kafka topic using the Confluent Cloud CLI, Java, or C/C++. The steps assume that:

  • Confluent CLI version 2 is being used. For more information, see Confluent CLI v2.
  • That you are already connected to your Confluent Cloud environment and cluster.

Audit Log cluster API key

Both the connector events feature and audit logging operate from the Audit Log cluster. There is a limit of two API keys for the Audit Log cluster. Enter the following Confluent CLI commands to find out how many API keys are being used for the Audit Log cluster.

  1. Get the Audit Log cluster ID.

    confluent connect event describe
    

    Example output:

    +-----------------+-------------------------------+
    | Cluster         | lkc-j3beid                    |
    | Environment     | env-o36079                    |
    | Service Account |                        ...... |
    | Topic Name      | confluent-connect-log-events  |
    +-----------------+-------------------------------+
    
  2. Check the API keys being used for the Audit Log cluster.

    confluent api-key list --resource <audit-log-cluster-ID>
    

    To create a new API and secret to consume connector events, you may need to delete an API key if there are two in use already. Make sure it’s okay to delete an API key first. Here is the command to delete an API key.

    confluent api-key delete <audit-log-API-key>
    

    After deleting an existing key, you can create the new audit log cluster API key.

    confluent api-key create --service-account <service-account-ID> --resource <audit-log-cluster-ID>
    
  3. Use the Audit Log API key.

    confluent api-key use <audit-log-API-key> --resource <audit-log-cluster-ID>
    

Consume events using the Confluent CLI

The following steps provide information for consuming event records from the Connect event topic using the Confluent CLI.

Prerequisites
  1. Enter the following command to get the event log topic name.

    confluent connect event describe
    

    Example output:

    +-----------------+---------------------------------+
    | Cluster         | <cluster-ID>                    |
    | Environment     | <environment-ID>                |
    | Service Account | <service-account-ID>            |
    | Topic Name      | confluent-connect-log-events    |
    +-----------------+---------------------------------+
    
  2. Enter the following command to begin consuming event records from the topic. When a Connect event occurs, it is provided as a JSON-formatted record in the displayed output.

    confluent environment use <environment-ID>
    confluent kafka cluster use <cluster-ID>
    confluent kafka topic consume -b confluent-connect-log-events
    

Consume events using Java and C/C++

Instructions for consuming records using Java and C/C++ are provided in the Confluent Cloud in-product documentation. To view the in-product documentation, go to the Administration menu and select Connect log events.

Confluent Cloud Event Log Admin Menu

Confluent Cloud Administration Menu