Edit and update a pipeline

Stream Designer enables editing and updating a running pipeline.

  • You can add components to the canvas while a pipeline is running.
  • You can’t modify activated components.
  • When you deactivate, you can select which topics to retain and which to delete.

Step 1: Create a pipeline project

A Stream Designer pipeline project defines all the components that are deployed for an application. In this step, you create a pipeline project and a canvas for designing the component graph.

  1. Log in to the Confluent Cloud Console and open the Cluster Overview page for the cluster you want to use for creating pipelines.

  2. In the navigation menu, click Stream Designer.

  3. Click Create pipeline, and in the Create new pipeline dialog, enter my-pipeline for the pipeline name.

    Stream Designer create new pipeline dialog in Confluent Cloud Console
  4. In the ksqlDB Cluster dropdown, select the ksqlDB cluster to use for your pipeline logic.

    Note

    If you don’t have a ksqlDB cluster yet, click Create new ksqlDB cluster to open the ksqlDB Cluster page and click Add cluster. When you’ve finished creating the ksqlDB cluster, return to the Create new pipeline dialog and click the refresh button to see your new ksqlDB cluster in the dropdown.

  5. Click Create pipeline.

    The Create a new pipeline page opens.

    Stream Designer create new pipeline page in Confluent Cloud Console
  6. Click Grant pipeline privileges.

  7. In the Grant activation privileges dialog, type “confirm” and click Confirm.

    The Activate pipeline and Deactivate pipeline buttons appear.

Step 2: Create a connector definition

Your pipeline starts with data produced by the Datagen source connector. In this step, you create a pipeline definition for a connector that produces mock pageview data to a Kafka topic.

  1. Click Start with connector and Start building.

    The Stream Designer canvas opens with a Source Connector component.

    Stream Designer initial view in Confluent Cloud Console
  1. In the Source Connector component, click Configure.

    The Source Connector page opens.

  2. In the search box, enter “datagen”.

    Stream Designer Datagen source connector search in Confluent Cloud Console
  3. Click the Datagen Source tile to open the Configuration page.

  4. In the Topic textbox, type “pageviews_topic”.

    Stream Designer Datagen source connector configuration in Confluent Cloud Console
  5. Click Continue to open Kafka credentials page.

    Stream Designer Datagen source connector configuration in Confluent Cloud Console
  6. Ensure that the the Global access tile is selected and click Generate a Kafka API key & secret to create the API key for the Datagen connector.

    A text file containing the newly generated API key and secret is downloaded to your local machine.

  7. Click Continue to configure the connector’s output.

  8. In the Select output record value format section, click JSON_SR, and in the Select a template section, click Pageviews.

    Stream Designer Datagen Source Connector configuration for mock pageviews data in Confluent Cloud Console
  9. Click Continue to open the Sizing page.

  10. In the Connector sizing section, leave the minimum number of tasks at 1.

  11. Click Continue to open the Review and launch page.

  12. In the Connector name textbox, enter “Datagen_pageviews” and click Continue.

    The Datagen source connector is configured and appears on the canvas with a corresponding topic component. The topic component is configured with the name you provided during connector configuration.

    Stream Designer Datagen Source Connector and topic in Confluent Cloud Console

Step 3: Configure the topic

For this step, the Datagen source connector produces to a topic with no corresponding stream.

  • Right-click the Stream component and select Remove.

    Stream Designer and stream context menu

    The pipeline is configured to produce pageview data from the Datagen source connector to the Kafka topic. In a later step, you register a stream on the topic while the pipeline is running.

    Stream Designer Connector and a topic component menu

    The pipeline is ready for activation.

Step 4: Activate the pipeline

  1. Click Activate to deploy the pipeline components.

    After a few seconds, the state of each component goes from Activating to Activated.

  2. Click the pageviews_topic component, and in the details page, click Messages.

    Your output should resemble the following.

    Stream Designer showing messages flowing in Confluent Cloud Console
  3. Close the pageviews_topic page.

    Even though the pipeline is running, and you can add components to the canvas and activate them later.

Step 5: Register a stream on the topic

  1. Hover over the Topic component and click the + icon that appears near the center.

    A context menu opens.

    Stream Designer and topic context menu in Confluent Cloud Console
  2. In the context menu, click Stream.

    A Stream component appears within the Topic component.

  3. In the Stream component, click Configure, and in the Stream configuration, name the stream “pageviews_stream”.

  4. In the Value Format dropdown, select JSON_SR.

    Stream Designer and stream config dialog in Confluent Cloud Console
  5. Click Save.

Step 6: Create a filter query

When your pipeline is activated and the Datagen Source connector is in the Running state, you can add new components to the canvas. You can’t change the configuration of activated components.

In the step, you add a filter query and a stream for the filtered data.

  1. Hover over the pageviews_stream component and click +.

    A context menu appears showing the components that accept a stream as input.

    Stream Designer and stream context menu
  2. In the context menu, click Filter.

    A Query component appears.

  3. In the Query component, click Configure and name the query “user 9 filter”. Click Save.

  4. In the Filter component, click Configure.

  5. In the Configuration dialog, name the filter “user 9”.

  6. In the Filter Expression field, enter the following SQL:

    userid = 'User_9'
    
    Stream Designer and filter configuration in Confluent Cloud Console
  7. Click Save to create the filter definition.

    The query component displays a red error triangle because it requires a stream, table, or another query component for its output. In the next step, you add a sink topic with a corresponding stream for the filter output.

Step 7: Create a filtered stream definition

The filter requires a stream, table, or another component for its output. The following steps show how to direct the filter’s output to a stream.

  1. Hover over the user 9 filter and click +.

    A context menu appears showing the components that accept a stream as input.

    Stream Designer and filter output context menu
  2. In the context menu, click Stream.

    A Topic component appears on the canvas.

  3. In the Topic component, click Configure. Name the topic “filtered_pageviews_topic” and click Save.

  4. In the Stream component, click Configure.

  5. In the Configuration dialog, name the stream “filtered_pageviews” and click Save.

Step 8: Activate the new components

The new components are configured but not activated.

Stream Designer showing filtered messages flowing in Confluent Cloud Console
  • Click Re-activate pipeline to activate the newly added components.

    Note

    If the filter component reports an error like Did not find any value schema for the topic, wait for the Datagen source connector to provision completely and activate the pipeline again.

Step 9: Inspect the pipeline

When all components are activated, your pipeline is deployed. In this step, you confirm that messages are flowing through the components you added sequentially.

  1. Click the filtered_pageviews topic, and in the details page, click Messages.

    Your output should resemble the following. Only User_9 records are shown.

    Stream Designer showing filtered messages flowing in Confluent Cloud Console

Step 10: Deactivate the pipeline

When you deactivate a pipeline, you have the option of retaining or deleting topics in the pipeline.

  1. Click Deactivate pipeline to delete all resources created by the pipeline.

    The Revert pipeline to draft? dialog appears. Click the dropdowns to delete or retain the listed topics. For this example, keep the Delete settings.

    Stream Designer showing the Revert dialog in Confluent Cloud Console
  2. Click Confirm and revert to draft to deactivate the pipeline and delete topics.

Step 11: Delete the pipeline

When all components have completed deactivation, you can delete the pipeline safely.

  1. Click the settings icon (settings-icon).

    The Pipeline Settings dialog opens.

    Stream Designer showing filtered messages flowing in Confluent Cloud Console
  2. Click Delete pipeline. In the Delete pipeline dialog, enter “confirm” and click Confirm.

  3. The pipeline and associated resources are deleted. You are returned to the Pipelines list.