Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

ccloud-stack Utility for Confluent Cloud

Overview

This ccloud-stack utility creates a stack of fully managed services in Confluent Cloud. It is a quick way to create fully managed resources in Confluent Cloud, which you can then use for learning and building other demos. The utility uses Confluent Cloud CLI under the hood to dynamically do the following in Confluent Cloud :

  • Create a new environment
  • Create a new service account
  • Create a new Kafka cluster and associated credentials
  • Enable Schema Registry and associated credentials
  • Create a new ksqlDB app and associated credentials
  • Create ACLs with wildcard for the service account
../../../../_images/ccloud-stack-resources1.png

In addition to creating these resources, ccloud-stack also generates a local configuration file with connection information to all of the above services. This file is particularly useful because it contains connection information to your Confluent Cloud instance, and any downstream application or Kafka client can use it, or you can use it for other demos or automation workflows.

Confluent Cloud Promo Code

The first 20 users to sign up for Confluent Cloud and use promo code C50INTEG will receive an additional $50 free usage (details).

Caution

This utility uses real Confluent Cloud resources. To avoid unexpected charges, carefully evaluate the cost of resources before launching the utility and ensure all resources are destroyed after you are done running it.

Prerequisites

Usage

Setup

  1. Clone the confluentinc/examples GitHub repository and check out the 5.5.15-post branch.

    git clone https://github.com/confluentinc/examples
cd examples
git checkout 5.5.15-post

  2. Change directory to the ccloud-stack utility:

    cd ccloud/ccloud-stack/

  3. Log in to Confluent Cloud with the command ccloud login, and use your Confluent Cloud username and password. The --save argument saves your Confluent Cloud user login credentials or refresh token (in the case of SSO) to the local netrc file.

    ccloud login --save

Create a ccloud-stack

  1. By default, the cloud-stack utility creates resources in the cloud provider aws in region us-west-2. If this is the target provider and region, create the stack by calling the bash script ccloud_stack_create.sh. For more options when configuring your ccloud-stack, see Advanced Options.

    ./ccloud_stack_create.sh

  2. In addition to creating all the resources in Confluent Cloud with associated service account and ACLs, running ccloud-stack also generates a local configuration file with all the Confluent Cloud connection information, which is useful for other demos/automation. View this file at stack-configs/java-service-account-<SERVICE_ACCOUNT_ID>.config. It resembles:

    # ------------------------------
# ENVIRONMENT ID: <ENVIRONMENT ID>
# SERVICE ACCOUNT ID: <SERVICE ACCOUNT ID>
# KAFKA CLUSTER ID: <KAFKA CLUSTER ID>
# SCHEMA REGISTRY CLUSTER ID: <SCHEMA REGISTRY CLUSTER ID>
# KSQLDB APP ID: <KSQLDB APP ID>
# ------------------------------
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
bootstrap.servers=<BROKER ENDPOINT>
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username\="<API KEY>" password\="<API SECRET>";
basic.auth.credentials.source=USER_INFO
schema.registry.basic.auth.user.info=<SR API KEY>:<SR API SECRET>
schema.registry.url=https://<SR ENDPOINT>
ksql.endpoint=<KSQLDB ENDPOINT>
ksql.basic.auth.user.info=<KSQLDB API KEY>:<KSQLDB API SECRET>

Destroy a ccloud-stack

  1. To destroy a cloud-stack created in the previous step, call the bash script ccloud_stack_destroy.sh and pass in the client properties file auto-generated in the step above.

    ./ccloud_stack_destroy.sh stack-configs/java-service-account-<SERVICE_ACCOUNT_ID>.config

Advanced Options

Select Cloud Provider and Region

By default, the cloud-stack utility creates resources in the cloud provider aws in region us-west-2. To create resources in another cloud provider or region other than the default, complete the following steps:

  1. View the available cloud providers and regions using the Confluent Cloud CLI:

    ccloud kafka region list

  2. Create the ccloud-stack and override the parameters CLUSTER_CLOUD and CLUSTER_REGION, as shown in the following example:

    CLUSTER_CLOUD=aws CLUSTER_REGION=us-west-2 ./ccloud_stack_create.sh

Use Existing Environment

By default, a new ccloud-stack creates a new environment. To reuse an existing environment, create the ccloud-stack and override the parameter ENVIRONMENT with an existing environment ID, as shown in the following example:

ENVIRONMENT=env-oxv5x ./ccloud_stack_create.sh

Automated Workflows

If you don’t want to create and destroy a ccloud-stack using the provided bash scripts ccloud_stack_create.sh and ccloud_stack_destroy.sh, you may pull in the ccloud_library and call the functions ccloud::create_ccloud_stack() and ccloud::destroy_ccloud_stack() directly.

  1. Get the ccloud_library:

    wget -O ccloud_library.sh https://raw.githubusercontent.com/confluentinc/examples/latest/utils/ccloud_library.sh

  2. Source the library

    source ./ccloud_library.sh

  3. Run the bash functions directly from the command line. To create the cloud-stack:

    CLUSTER_CLOUD=aws
CLUSTER_REGION=us-west-2
ccloud::create_ccloud_stack

    To destroy the ccloud-stack:

    ccloud::destroy_ccloud_stack

Additional Resources