Confluent Cloud Connector Data Previews¶
You can sample the output from a connector before you launch it. This allows you to add to or correct configuration properties before your connector actually runs in your environment.
The following are current limitations for connector data previews:
- Data previews are only available for certain Confluent Cloud connectors.
- Data previews are not available using the Confluent CLI or API.
- There is a limit of seven data preview requests per day, per organization.
Launch the preview¶
Once you have configured a connector, you are provided with a Data preview button that creates a data preview request with this connector configuration.
Set the number of tasks configuration property to
1 when you run the
data preview request. If needed, change the task number after validating the
data preview output.
If using granular access keys, for the data preview to work please ensure that
the service account has
WRITE access to the topic prefix
Similarly, to view the data preview output the logged-in user needs to have
to the topic prefix
- It may take a few minutes for the output to generate.
- If the source system does not produce data, the preview will expire without producing any data. Make sure the source system is ready to produce data before creating the data preview request.
- Occasionally, the data preview request may timeout before the required resources are provisioned. When this happens the error message
Connector failed to provisionis displayed. You can retry the preview request or launch the actual connector.
- Previewed records are stored in a separate data preview output topic named
data-preview-ldpc-<id>(in your Kafka cluster). This topic is deleted when you click the delete icon next to the preview listing.
Once it starts, the Confluent Cloud Console displays a screen similar to the one shown below.
As soon as the preview is triggered, the data preview request will have the status
Generating preview. Once the data preview output topic is created this status changes to
Preview available, which is when you can click the request to see the request’s output.
If there is an error in output generation, the data preview request status will be
Preview failed. You can click the request to look at the error message to take action on.
If the data preview request is unable to find any data in the external system, the request status will eventually change to
Please note that the data preview request list shows ten most recent requests and there is a limit of ten concurrent requests.
View the data preview output¶
The Cloud Console displays a preview of the output based on the properties you specified when setting up the connector. You can expand a record from the running preview to validate the output.
The expanded preview record has two sections:
The metadata section shows information about how the record is produced. This
section is only shown in data preview output. This is not part of the actual data
ingested by the connector.
The record section shows the actual data that is sent to the topic when the connector is running.
- Preview data is not converted to any specified format (AVRO, JSON_SR, etc.)
- Some connectors and configurations output data as bytes. For such connectors the data preview will have the preview data in
valuefield of the preview output record.
- Data is never ingested into the designated Kafka topic configured in the connector. Data preview output is maintained only in the data preview output topic and formatted as JSON.
- Data preview requests read actual data from the source system. Consuming data from some source systems can be a destructive operation. You can use a test source in these situations.
- The data preview output view shows at most twenty records. If source system has large amounts of data, it is possible that the request generated more than twenty records. In such cases, if you’re interested to view the full output topic you can do so via the Topics tab.
Edit the configuration¶
If you don’t see what you want in the data preview output, click Edit Configuration and add properties, remove properties, or modify properties until you have the data preview output you expect. If the data preview output looks good, you can create the connector after re-entering your credentials. Note that if you do not make any configuration changes, the Data preview button is inactive on the connector launch screen.
Delete the preview¶
The data preview output topic uses actual resources. After the data preview request has
completed, a delete button appears in the listing. Before launching the connector, be sure to click this button
to delete the output topic. This will also change the data preview request status to