Quick Start for Stream Designer on Confluent Cloud

The following steps show how to create a pipeline in Stream Designer.

Prerequisites

To use Stream Designer for creating pipelines, the following prerequisites must be in place.

• A Schema Registry cluster
• A ksqlDB cluster
• Sufficient permissions to create pipelines

Step 1: Create a pipeline project

A Stream Designer pipeline project defines all the components that are deployed for an application. In this step, you create a pipeline project and a canvas for designing the component graph.

1. Log in to the Confluent Cloud Console and open the Cluster Overview page for the cluster you want to use for creating pipelines.

2. In the navigation menu, click Stream Designer.

3. Click Create pipeline.

   The Create a new pipeline page opens.

   

Step 2: Create a connector definition

Your pipeline starts with data produced by the Datagen source connector. In this step, you create a pipeline definition for a connector that produces mock pageviews data to a Kafka topic.

1. Click Start with Connector and then click Start building.

   The Stream Designer canvas opens, with the Source Connector details view visible.

   

2. In the Source Connector page click the Datagen Source tile to open the Configuration page.

3. In the Topic textbox, type “pageviews_topic”.

   

4. Click Continue to open Kafka credentials page.

   

5. Ensure that the the Global access tile is selected and click Generate API key & download to create the API key for the Datagen connector.

   A text file containing the newly generated API key and secret is downloaded to your local machine.

6. Click Continue to configure the connector’s output.

7. In the Select output record value format section, click JSON_SR, and in the Select a template section, click Pageviews.

   

8. Click Continue to open the Sizing page.

9. In the Connector sizing section, leave the minimum number of tasks at 1.

10. Click Continue to open the Review and launch page.

11. In the Connector name textbox, enter “Datagen_pageviews” and click Continue.

    The Datagen source connector is configured and appears on the canvas with a corresponding topic component. The topic component is configured with the name you provided during connector configuration. Also, a stream is registered on the topic.

    

Step 3: Register a stream on the topic

Stream Designer enables registering a stream on an underlying topic.

1. In the Stream component, click Configure to open the stream configuration dialog.

2. In the Name textbox, enter “pageviews_stream”.

3. In the Value Format dropdown, select JSON_SR.

   

4. Click Save.

   The Topic component updates with the stream name.

   

Step 4: Create a transformation definition

Stream Designer enables defining transformations on your streams. In this step, you create the definition for a filter component and specify its behavior by using a simple SQL statement.

1. Hover over the pageviews_stream component and click +.

   A context menu appears showing the components that accept a stream as input.

   

2. In the context menu, click Filter.

   A Query component appears.

3. In the Query component, click Configure.

   The Configuration dialog opens.

4. In the Filter Expression field, enter the following SQL:

   userid = 'User_9'
   

   

5. Click Save to create the filter definition.

   The query component displays a red error triangle because it requires a stream, table, or another query component for its output. In the next step, you add a sink topic with a corresponding stream for the filter output.

Step 5: Create a filtered stream definition

The filter requires a stream, table, or another component for its output. The following steps show how to direct the filter’s output to a stream.

1. Hover over the user 9 filter and click +.

   A context menu appears showing the components that accept a stream as input.

   

2. In the context menu, click Stream.

   A Topic component appears on the canvas and the stream configuration dialog opens.

3. In the Configuration dialog, name the stream “filtered_pageviews” and click Save.

Your pipeline is ready for activation. The design canvas should resemble the following.

Step 6: Activate the pipeline

In this step, you enable security for the pipeline and activate it.

1. Click Activate to deploy the pipeline components.

   The Pipeline activation dialog opens.

2. In the ksqlDB Cluster dropdown, select the ksqlDB cluster to use for your pipeline logic.

   Note

   If you don’t have a ksqlDB cluster yet, click Create new ksqlDB cluster to open the ksqlDB Cluster page and then click Add cluster. When you’ve finished creating the ksqlDB cluster, return to the Create new pipeline dialog and click the refresh button to see your new ksqlDB cluster in the dropdown.

3. In the Activation privileges section, click Grant privileges.

4. Click Confirm to activate your pipeline.

   After a few seconds, the state of each component goes from Activating to Activated.

   Note

   If the filter component reports an error like Did not find any value schema for the topic, wait for the Datagen source connector to provision completely and activate the pipeline again.

When all components are activated, your pipeline is deployed.

Step 7: Inspect the pipeline

In this step, you confirm that messages are flowing from the source connector through all of the components to the sink topic.

1. Click pageviews_topic, and in the details page, click Messages.

   Your output should resemble the following.

   

2. Close the pageviews_topic page.

3. Click the filtered_pageviews topic, and in the details page, click Messages.

   Your output should resemble the following. Only User_9 records are shown.

   

Step 8: Deactivate the pipeline

To avoid incurring costs, deactivate the pipeline to delete all resources created by the pipeline.

When you deactivate a pipeline, you have the option of retaining or deleting topics in the pipeline.

1. Click the settings icon ().

   The Pipeline Settings dialog opens.

2. Click Deactivate pipeline to delete all resources created by the pipeline.

   The Revert pipeline to draft? dialog appears. Click the dropdowns to delete or retain the listed topics. For this example, keep the Delete settings.

   

3. Click Confirm and revert to draft to deactivate the pipeline and delete topics.

Step 9: Delete the pipeline

When all components have completed deactivation, you can delete the pipeline safely.

1. Click the settings icon.

   The Pipeline Settings dialog opens.

   

2. Click Delete pipeline. In the Delete pipeline dialog, enter “confirm” and click Confirm.

3. The pipeline and associated resources are deleted. You are returned to the Pipelines list.

Next steps

• Create a Join Pipeline for Stream Designer on Confluent Cloud
• Create an Aggregation Pipeline for Stream Designer on Confluent Cloud
• Manage Pipelines With the CLI for Stream Designer on Confluent Cloud
• Manage Pipelines With the REST API for Stream Designer on Confluent Cloud

Suggested Resources

• Blog post: Introducing Stream Designer: The Visual Builder for Streaming Data Pipelines
• Video: Stream Designer: The Visual Builder for Kafka Pipelines in Confluent Cloud