Manage pipeline life cycle by using the Confluent CLI

You can use the Confluent CLI to create and manage Stream Designer pipelines. Use the following commands to perform operations on your pipelines.

Prequisites for using CLI pipeline commands

To prepare for managing pipelines in Confluent Cloud, follow these steps.

  1. Request authorization from an OrganizationAdmin to create and modify pipelines in your Confluent Cloud environment.
  2. Log in to Confluent Cloud by using the confluent login command.
  3. Specify the Kafka cluster where the pipeline runs by using the confluent kafka cluster use command.
  4. Get an identifier for the ksqlDB cluster by using the confluent ksql cluster list command.

Get the Kafka cluster ID

You must specify the Kafka cluster that hosts your pipeline. Run the following commands to get the cluster ID and assign it as the active cluster.

  1. List the Kafka clusters in your environment.

    confluent kafka cluster list
    

    Your output should resemble:

           Id      |     Name     | Type  | Provider |  Region   | Availability | Status
    ---------------+--------------+-------+----------+-----------+--------------+---------
        lkc-9z8y7x | test-cluster | BASIC | gcp      | us-west4  | single-zone  | UP
    
  2. Specify the Kafka cluster for your pipeline.

    confluent kafka cluster use lkc-9z8y7x
    

    Your output should resemble:

    Set Kafka cluster "lkc-9z8y7x" as the active cluster for environment "env-e5d4c3".
    

Get the ksqlDB cluster ID

You need the ID for the ksqlDB cluster that runs your pipeline’s logic. Run the following command to get the ksqlDB cluster ID.

  • List the ksqlDB clusters in your environment.

    confluent ksql cluster list
    

    Your output should resemble:

           ID       |        Name        | Topic Prefix |   Kafka    |
    ----------------+--------------------+--------------+------------+ ...
      lksqlc-3c2b1a | ksqlDB_cluster_0   | pksqlc-b0b03 | lkc-9z8y7x |
    

    Note the ID for the ksqlDB cluster that you want to use for your pipeline, as you will need it for the pipeline CLI commands. In this example, the ID is lksqlc-3c2b1a.

Create a new pipeline

Run the confluent pipeline create command to create a new pipeline in the specified ksqlDB cluster. When the pipeline is created, it doesn’t have any configuration for logic or data sources and sinks. You specify these details in the Stream Designer UI in Confluent Cloud Console.

  1. Run the following command to create a pipeline in the ksqlDB cluster that has the ID <ksqldb-cluster-id>.

    confluent pipeline create --name "test-pipeline" \
                              --description "first CLI pipeline" \
                              --ksql-cluster <ksqldb-cluster-id>
    

    Your output should resemble:

    +--------------+--------------------------------+
    | ID           | pipe-1a2b3c                    |
    | Name         | test-pipeline                  |
    | Description  | first CLI pipeline             |
    | KSQL Cluster | lksqlc-3c2b1a                  |
    | State        | draft                          |
    | Created At   | 2022-10-03 20:03:54.531015508  |
    |              | +0000 UTC                      |
    | Updated At   | 2022-10-03 20:03:54.531015508  |
    |              | +0000 UTC                      |
    +--------------+--------------------------------+
    

    Note the pipeline ID, which you need for other CLI commands.

    In the Confluent Cloud Console, you can view the pipeline that you created.

  2. Open Confluent Cloud Console, and in the navigation menu, click Stream Designer to open the Pipelines page.

    Stream Designer showing the pipelines list in Confluent Cloud Console

    Use Stream Designer to build your pipeline. For more information, see Quick Start for Stream Designer.

View existing pipelines

In Confluent CLI, run the confluent pipeline list command to view the pipelines that you’ve created.

confluent pipeline list

Your output should resemble:

      ID      |     Name      |    Description     | KSQL Cluster  | State
--------------+---------------+--------------------+---------------+--------
  pipe-1a2b3c | test-pipeline | first CLI pipeline | lksqlc-v7o935 | draft

Activate a pipeline

When your pipeline is ready to activate, use the confluent pipeline activate command to deploy it.

Activation means that topics are created, connectors are deployed, and ksqlDB persistent queries are created.

  1. Run the following command to activate a pipeline.

    confluent pipeline activate --id <pipeline-id>
    

    Your output should resemble:

    +--------------+--------------------------------+
    | ID           | pipe-1a2b3c                    |
    | Name         | test-pipeline                  |
    | Description  | first CLI pipeline             |
    | KSQL Cluster | lksqlc-3c2b1a                  |
    | State        | activating                     |
    | Created At   | 2022-10-03 20:03:54.531015508  |
    |              | +0000 UTC                      |
    | Updated At   | 2022-10-03 20:33:29.961081381  |
    |              | +0000 UTC                      |
    +--------------+--------------------------------+
    
  2. After a few minutes, run the confluent pipeline list command to check on your pipeline’s activation status. If there were no activation errors, your output should resemble:

          ID      |     Name      |    Description     | KSQL Cluster  | State
    --------------+---------------+--------------------+---------------+---------
      pipe-1a2b3c | test-pipeline | first CLI pipeline | lksqlc-3c2b1a | active
    

Update an existing pipeline

Use the confluent pipeline update command to update the name and description of a pipeline

  1. Run the following command to update a pipeline.

    confluent pipeline update <pipeline-id> --name <pipeline-name> --description "<pipeline-description>"
    

    Your output should resemble:

    +--------------+--------------------------------+
    | ID           | pipe-1a2b3c                    |
    | Name         | test-pipeline-2                |
    | Description  | Renamed pipeline               |
    | KSQL Cluster | lksqlc-3c2b1a                  |
    | State        | active                         |
    | Created At   | 2022-10-03 20:03:54.531015508  |
    |              | +0000 UTC                      |
    | Updated At   | 2022-10-03 21:02:31.806605354  |
    |              | +0000 UTC                      |
    +--------------+--------------------------------+
    
  2. Run the confluent pipeline list command to confirm your changes.

    Your output should resemble:

          ID      |      Name       |   Description    | KSQL Cluster  | State
    --------------+-----------------+------------------+---------------+---------
      pipe-1a2b3c | test-pipeline-2 | Renamed pipeline | lksqlc-3c2b1a | active
    

Describe an existing pipeline

Use the confluent pipeline describe command to view details of a pipeline.

  • Run the following command to describe a pipeline.

    confluent pipeline describe <pipeline-id>
    

    Your output should resemble:

    +--------------+--------------------------------+
    | ID           | pipe-1a2b3c                    |
    | Name         | test-pipeline-2                |
    | Description  | Renamed pipeline               |
    | KSQL Cluster | lksqlc-3c2b1a                  |
    | State        | active                         |
    | Created At   | 2022-10-03 20:03:54.531015508  |
    |              | +0000 UTC                      |
    | Updated At   | 2022-10-03 21:02:31.806605354  |
    |              | +0000 UTC                      |
    +--------------+--------------------------------+
    

Deactivate a running pipeline

Use the confluent pipeline deactivate command to deactivate a running pipeline. You can specify the topics to be retained when the pipeline deactivates by using the --retained-topics flag.

  1. Run the following command to activate a pipeline.

    confluent pipeline deactivate <pipeline-id>
    

    Your output should resemble:

    +--------------+--------------------------------+
    | ID           | pipe-1a2b3c                    |
    | Name         | test-pipeline-2                |
    | Description  | Renamed pipeline               |
    | KSQL Cluster | lksqlc-3c2b1a                  |
    | State        | deactivating                   |
    | Created At   | 2022-10-03 20:03:54.531015508  |
    |              | +0000 UTC                      |
    | Updated At   | 2022-10-03 21:02:32.119654775  |
    |              | +0000 UTC                      |
    +--------------+--------------------------------+
    
  2. Run the confluent pipeline list command to confirm the pipeline is deactivated and back in the draft state.

    Your output should resemble:

          ID      |      Name       |   Description    | KSQL Cluster  | State
    --------------+-----------------+------------------+---------------+---------
      pipe-1a2b3c | test-pipeline-2 | Renamed pipeline | lksqlc-3c2b1a | draft
    

Delete a pipeline

Use the confluent pipeline delete command to remove a pipeline from Confluent Cloud.

Note

Deactivate the pipeline before deletion. If you don’t deactivate first, Confluent Cloud resources, like connectors and topics, are not deleted.

  1. Run the following command to activate a pipeline.

    confluent pipeline deactivate <pipeline-id>
    

    Your output should resemble:

    Requested to delete pipeline "pipe-1a2b3c".
    
  2. Run the confluent pipeline describe command to confirm the pipeline is deleted.

confluent pipeline describe <pipeline-id>

Your output should resemble:

::

   +--------------+--------------------------------+
   | ID           | pipe-1a2b3c                    |
   | Name         | test-pipeline-2                |
   | Description  | Renamed pipeline               |
   | KSQL Cluster | lksqlc-3c2b1a                  |
   | State        | deleted                        |
   | Created At   | 2022-10-03 20:03:54.531015508  |
   |              | +0000 UTC                      |
   | Updated At   | 2022-10-03 21:22:43.202347667  |
   |              | +0000 UTC                      |
   +--------------+--------------------------------+

Suggested Resources