Preview Connector Output

You can sample the output from a connector before you launch it. This allows you to add to or correct configuration properties before your connector actually runs in your environment.

Note

The following are current limitations for previewing connector output:

  • Previewing output is only available for Basic and Standard clusters.
  • Previewing output is only available for certain Confluent Cloud connectors.
  • Previewing output is not available using the Confluent Cloud CLI or API.
  • There is a limit of seven preview output requests per day, per organization.

Launch the preview

Once you have configured and launched a connector, you are provided with a Data preview button that launches the connector in preview output mode.

Important

Set the number of tasks configuration property to 1 when you run the preview request. If needed, change the task number after validating the preview output.

Output Preview Launch Button

Data preview and Launch

Note

  • It may take a few minutes for the output to generate.
  • If the source system does not produce data, the preview will expire without producing any data. Make sure the source system is ready to produce data before launching the connector in preview mode.
  • Occasionally, the preview request may timeout before the required resources are provisioned. When this happens the error message "Connector failed to provision" is displayed. You can retry the preview request or launch the actual connector.
  • Previewed records are stored in a separate preview output topic named data-preview-ldpc-<id> (in your Kafka cluster). This topic is deleted when you click the delete icon next to the preview listing.

Once it starts, the Confluent Cloud Console displays a screen similar to the one shown below.

Output Preview Request Selection

Data preview requests

Click and launch the preview request you want to view.

View the data output

The Cloud Console displays a preview of the output based on the properties you specified when setting up the connector. You can expand a record from the running preview to validate the output.

The expanded preview record has two sections: "metadata" and "record". The metadata section shows information about how the record is produced. This section is only shown in data preview mode. This is not part of the actual data ingested by the connector.

Running Output Metadata Section

Metadata section

The record section shows the actual data that is sent to the topic when the connector is running.

Note

  • Preview data is not converted to any specified format (AVRO, JSON_SR, etc.)
  • Data is never ingested into a designated Kafka topic. Preview data is maintained only in the preview topic and formatted as JSON.
  • Preview requests read actual data from the source system. Consuming data from some source systems can be a destructive operation. You can use a test source in these situations.
Running Output Record Section

Record section

Edit the configuration

If you don’t see what you want in the running connector preview, click Edit Configuration and add properties, remove properties, or modify properties until you have the data output you expect. If the preview output looks good, you can create the connector after re-entering your credentials. Note that if you do not make any configuration changes, the Data preview button is inactive on the connector launch screen.

Edit Configuration

Edit configuration

Delete the preview

The preview output topic uses actual resources. After the preview request has completed, a delete button appears in the listing. Be sure to click this button to delete the output topic before launching your connector.

Delete Preview

Delete preview