Confluent Cloud Connector Data Previews

You can sample the output from a connector before you launch it. This allows you to add to or correct configuration properties before your connector actually runs in your environment.

Limitations

The following are current limitations for connector data previews:

  • Data previews are only available for certain Confluent Cloud connectors.
  • Data previews are not available using the Confluent CLI or API.
  • There is a limit of seven data preview requests per day, per organization.

Launch the preview

Once you have configured a connector, you are provided with a Data preview button that creates a data preview request with this connector configuration.

Output Preview Launch Button

Data preview and Launch

Note the following details when using this feature:

  • Set the number of tasks configuration property to 1 when you run the data preview request. If needed, change the task number after validating the data preview output.
  • If using granular access keys, for the data preview to work you must ensure that the service account has CREATE and WRITE access to the topic prefix data-preview. Similarly, to view the data preview output the logged-in user needs to have READ access to the topic prefix data-preview.
  • It may take a few minutes for the output to generate.
  • If the source system does not produce data, the preview will expire without producing any data. Make sure the source system is ready to produce data before creating the data preview request.
  • Occasionally, the data preview request may timeout before the required resources are provisioned. When this happens the error message Connector failed to provision is displayed. You can retry the preview request or launch the actual connector.
  • Previewed records are stored in a separate data preview output topic named data-preview-ldpc-<id> (in your Kafka cluster). This topic is deleted when you click the delete icon next to the preview listing.

After you press the button, the Confluent Cloud Console displays a screen similar to the one shown below.

Output Preview Request Selection

Data preview requests

As soon as the preview is triggered, the data preview request will have the status Generating preview. Once the data preview output topic is created this status changes to Preview available, which is when you can click the request to see the request’s output. If there is an error in output generation, the data preview request status will be Preview failed. You can click the request to look at the error message to take action on. If the data preview request is unable to find any data in the external system, the request status will eventually change to Preview expired.

Note that the data preview request list shows ten most recent requests and there is a limit of ten concurrent requests.

View the data preview output

The Cloud Console displays a preview of the output based on the properties you specified when setting up the connector. You can expand a record from the running preview to validate the output.

The expanded preview record has two sections: metadata and record. The metadata section shows information about how the record is produced. This section is only shown in data preview output. This is not part of the actual data ingested by the connector.

Running Output Metadata Section

Metadata section

The record section shows the actual data that is sent to the topic when the connector is running. Note the following additional details when viewing preview data:

  • Preview data is not converted to any specified format (AVRO, JSON_SR, etc.)
  • Some connectors and configurations output data as bytes. For such connectors the data preview will have the preview data in base64 in the value field of the preview output record.
  • Data is never ingested into the designated Kafka topic configured in the connector. Data preview output is maintained only in the data preview output topic and formatted as JSON.
  • Data preview requests read actual data from the source system. Consuming data from some source systems can be a destructive operation. You can use a test source in these situations.
  • The data preview output view shows at most twenty records. If source system has large amounts of data, it is possible that the request generated more than twenty records. In such cases, if you’re interested to view the full output topic you can do so via the Topics tab.
Running Output Record Section

Record section

Edit the configuration

If you don’t see what you want in the data preview output, click Edit Configuration and add properties, remove properties, or modify properties until you have the data preview output you expect. If the data preview output looks good, you can create the connector after re-entering your credentials. Note that if you do not make any configuration changes, the Data preview button is inactive on the connector launch screen.

Edit Configuration

Edit configuration

Delete the preview

The data preview output topic uses actual resources. After the data preview request has completed, a delete button appears in the listing. Before launching the connector, be sure to click this button to delete the output topic. This will also change the data preview request status to Preview expired.

Delete Preview

Delete preview