Edit and Update a Pipeline for Stream Designer on Confluent Cloud

Stream Designer enables editing and updating a running pipeline.

• You can add components to the canvas while a pipeline is running.
• You can’t modify activated components.
• When you deactivate, you can select which topics to retain and which to delete.

Step 1: Create a pipeline project

A Stream Designer pipeline project defines all the components that are deployed for an application. In this step, you create a pipeline project and a canvas for designing the component graph.

1. Log in to the Confluent Cloud Console and open the Cluster Overview page for the cluster you want to use for creating pipelines.

2. In the navigation menu, click Stream Designer.

3. Click Create pipeline.

   The Create a new pipeline page opens.

   

Step 2: Create a connector definition

Your pipeline starts with data produced by the Datagen source connector. In this step, you create a pipeline definition for a connector that produces mock pageview data to a Kafka topic.

1. Click Start with connector and Start building.

   The Stream Designer canvas opens with a Source Connector component.

   

1. Click Start with Connector and then click Start building.

   The Stream Designer canvas opens, with the Source Connector details view visible.

   

2. In the Source Connector page click the Datagen Source tile to open the Configuration page.

3. In the Topic textbox, type “pageviews_topic”.

   

4. Click Continue to open Kafka credentials page.

   

5. Ensure that the the Global access tile is selected and click Generate API key & download to create the API key for the Datagen connector.

   A text file containing the newly generated API key and secret is downloaded to your local machine.

6. Click Continue to configure the connector’s output.

7. In the Select output record value format section, click JSON_SR, and in the Select a template section, click Pageviews.

   

8. Click Continue to open the Sizing page.

9. In the Connector sizing section, leave the minimum number of tasks at 1.

10. Click Continue to open the Review and launch page.

11. In the Connector name textbox, enter “Datagen_pageviews” and click Continue.

    The Datagen source connector is configured and appears on the canvas with a corresponding topic component. The topic component is configured with the name you provided during connector configuration. Also, a stream is registered on the topic.

    

Step 3: Configure the topic

For this step, the Datagen source connector produces to a topic with no corresponding stream.

• Right-click the Stream component and select Remove.

  

  The pipeline is configured to produce pageview data from the Datagen source connector to the Kafka topic. In a later step, you register a stream on the topic while the pipeline is running.

  

  The pipeline is ready for activation.

Step 4: Activate the pipeline

In this step, you enable security for the pipeline and activate it.

1. Click Activate to deploy the pipeline components.

   The Pipeline activation dialog opens.

2. In the ksqlDB Cluster dropdown, select the ksqlDB cluster to use for your pipeline logic.

   Note

   If you don’t have a ksqlDB cluster yet, click Create new ksqlDB cluster to open the ksqlDB Cluster page and then click Add cluster. When you’ve finished creating the ksqlDB cluster, return to the Create new pipeline dialog and click the refresh button to see your new ksqlDB cluster in the dropdown.

3. In the Activation privileges section, click Grant privileges.

4. Click Confirm to activate your pipeline.

   After a few seconds, the state of each component goes from Activating to Activated.

   Note

   If the filter component reports an error like Did not find any value schema for the topic, wait for the Datagen source connector to provision completely and activate the pipeline again.

5. Click the pageviews_topic component, and in the details page, click Messages.

   Your output should resemble the following.

   

6. Close the pageviews_topic page.

   Even though the pipeline is running, and you can add components to the canvas and activate them later.

Step 5: Register a stream on the topic

1. Hover over the Topic component and click the + icon that appears near the center.

   A context menu opens.

   

2. In the context menu, click Stream.

   A Stream component appears within the Topic component.

3. In the Stream component, click Configure, and in the Stream configuration, name the stream “pageviews_stream”.

4. In the Value Format dropdown, select JSON_SR.

   

5. Click Save.

Step 6: Create a filter query

When your pipeline is activated and the Datagen Source connector is in the Running state, you can add new components to the canvas. You can’t change the configuration of activated components.

In the step, you add a filter query and a stream for the filtered data.

1. Hover over the pageviews_stream component and click +.

   A context menu appears showing the components that accept a stream as input.

   

2. In the context menu, click Filter.

   A Query component appears.

3. In the Query component, click Configure.

   The Configuration dialog opens.

4. In the Filter Expression field, enter the following SQL:

   userid = 'User_9'
   

   

5. Click Save to create the filter definition.

   The query component displays a red error triangle because it requires a stream, table, or another query component for its output. In the next step, you add a sink topic with a corresponding stream for the filter output.

Step 7: Create a filtered stream definition

The filter requires a stream, table, or another component for its output. The following steps show how to direct the filter’s output to a stream.

1. Hover over the user 9 filter and click +.

   A context menu appears showing the components that accept a stream as input.

   

2. In the context menu, click Stream.

   A Topic component appears on the canvas and the stream configuration dialog opens.

3. In the Configuration dialog, name the stream “filtered_pageviews” and click Save.

Step 8: Activate the new components

The new components are configured but not activated.

• Click Re-activate pipeline to activate the newly added components.

  Note

  If the filter component reports an error like Did not find any value schema for the topic, wait for the Datagen source connector to provision completely and activate the pipeline again.

Step 9: Inspect the pipeline

When all components are activated, your pipeline is deployed. In this step, you confirm that messages are flowing through the components you added sequentially.

1. Click the filtered_pageviews topic, and in the details page, click Messages.

   Your output should resemble the following. Only User_9 records are shown.

   

Step 10: Deactivate the pipeline

When you deactivate a pipeline, you have the option of retaining or deleting topics in the pipeline.

1. Click the settings icon ().

   The Pipeline Settings dialog opens.

2. Click Deactivate pipeline to delete all resources created by the pipeline.

   The Revert pipeline to draft? dialog appears. Click the dropdowns to delete or retain the listed topics. For this example, keep the Delete settings.

   

3. Click Confirm and revert to draft to deactivate the pipeline and delete topics.

Step 11: Delete the pipeline

When all components have completed deactivation, you can delete the pipeline safely.

1. Click the settings icon.

   The Pipeline Settings dialog opens.

   

2. Click Delete pipeline. In the Delete pipeline dialog, enter “confirm” and click Confirm.

3. The pipeline and associated resources are deleted. You are returned to the Pipelines list.