# Example: Create Fully Managed Services on Confluent Cloud This tutorial creates a set of fully managed Apache Kafka® services in Confluent Cloud by using the `ccloud-stack` utility. It is a quick way to create resources in Confluent Cloud with correct credentials and permissions. This can be useful as a starting point that you can then use for learning, extending, and building other examples. The utility uses the Confluent CLI to dynamically perform the following in Confluent Cloud: - Create a new environment - Create a new service account - Create a new Kafka cluster and associated credentials - Enable Schema Registry and associated credentials - Create role binding for the service account In addition to creating these resources, `ccloud-stack` also generates a local configuration file with connection information to all of the above services. This file is particularly useful because it contains connection information to your Confluent Cloud instance, and any downstream application or Kafka client can use it, or you can use it for other demos or automation workflows. ## Cost to Run ccloud-stack ### Caution Confluent Cloud examples that use actual Confluent Cloud resources might be billable. An example might create a new Confluent Cloud environment, Kafka cluster, topics, ACLs, service accounts, or resources that have hourly charges like connectors and ksqlDB applications. To avoid unexpected charges, carefully [evaluate the cost of resources](../../billing/overview.md#cloud-billing) before you start. After you are done running a Confluent Cloud example, destroy all Confluent Cloud resources to avoid accruing hourly charges for services and verify that they have been deleted. This utility uses real Confluent Cloud resources. It is intended to be a quick way to create resources in Confluent Cloud with correct credentials and permissions, useful as a starting point from which you can then use for learning, extending, and building other examples. - If you just run `ccloud-stack` without explicitly enabling Confluent Cloud ksqlDB, then there is no billing charge until you create a topic, produce data to the Kafka cluster, or provision any other fully managed service. - If you run `ccloud-stack` with enabling Confluent Cloud ksqlDB (1 CSU), then you will begin to accrue charges immediately. Here is a list of Confluent CLI commands issued by the utility that create resources in Confluent Cloud (function `ccloud::create_ccloud_stack()` source code is in [ccloud_library](https://github.com/confluentinc/examples/tree/latest/utils/ccloud_library.sh)). By default, the Confluent Cloud ksqlDB app is not created with `ccloud-stack`, you have to explicitly enable it. ```text confluent iam service-account create $SERVICE_NAME --description "SA for $EXAMPLE run by $CLOUD_EMAIL" -o json confluent environment create $ENVIRONMENT_NAME -o json confluent kafka cluster create "$CLUSTER_NAME" --cloud $CLUSTER_CLOUD --region $CLUSTER_REGION confluent api-key create --service-account $SERVICE_ACCOUNT_ID --resource $RESOURCE -o json // for kafka confluent iam rbac role-binding create --principal User:$SERVICE_ACCOUNT_ID --role EnvironmentAdmin --environment $ENVIRONMENT -o json ``` ## Prerequisites - Create a user account in [Confluent Cloud](https://www.confluent.io/confluent-cloud/) - Local install of the [Confluent CLI](https://docs.confluent.io/confluent-cli/current/install.html) v3.0.0 or later. - `jq` tool Note that `ccloud-stack` has been validated on macOS 10.15.3 with bash version 3.2.57. If you encounter issues on any other operating systems or versions, please open a GitHub issue at [confluentinc/examples](https://github.com/confluentinc/examples/issues). ## Usage ### Setup 1. Clone the [confluentinc/examples](https://github.com/confluentinc/examples) GitHub repository and check out the `master` branch. ```bash git clone https://github.com/confluentinc/examples cd examples git checkout master ``` 2. Change directory to the ccloud-stack utility: ```bash cd ccloud/ccloud-stack/ ``` 3. Log in to Confluent Cloud with the command `confluent login`, and use your Confluent Cloud username and password. The `--save` argument saves your Confluent Cloud user login credentials or refresh token (in the case of SSO) locally. ```shell confluent login --save ``` ### Create a ccloud-stack 1. By default, the `cloud-stack` utility creates resources in the cloud provider `aws` in region `us-west-2`. If this is the target provider and region, create the stack by calling the bash script [ccloud_stack_create.sh](https://github.com/confluentinc/examples/tree/latest/ccloud/ccloud-stack/ccloud_stack_create.sh). For more options when configuring your `ccloud-stack`, see [Advanced Options](#ccloud-stack-options). ```bash ./ccloud_stack_create.sh ``` 2. You will be prompted twice. Note the second prompt which is where you can optionally enable Confluent Cloud ksqlDB. ```text Do you still want to run this script? [y/n] y Do you also want to create a Confluent Cloud ksqlDB app (hourly charges may apply)? [y/n] n ``` 3. `ccloud-stack` assigns the [EnvironmentAdmin](https://docs.confluent.io/cloud/current/access-management/access-control/cloud-rbac.html#environmentadmin) role to the service account it creates. This permissive role is useful for development and learning environments. In production, configure a stricter role and potentially use ACLs with RBAC as documented [here](https://docs.confluent.io/cloud/current/access-management/access-control/cloud-rbac.html#use-acls-with-rbac). 4. In addition to creating all of the resources in Confluent Cloud with an associated service account, running `ccloud-stack` also generates a local configuration file with Confluent Cloud connection information, which is useful for creating demos or additional automation. View this file at `stack-configs/java-service-account-.config`. It resembles: ```text # ------------------------------ # ENVIRONMENT_ID: # SERVICE_ACCOUNT_ID: # KAFKA_CLUSTER_ID: # SCHEMA_REGISTRY_CLUSTER_ID: # KSQLDB_APP_ID: # ------------------------------ sasl.mechanism=PLAIN security.protocol=SASL_SSL bootstrap.servers= sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='' password=''; basic.auth.credentials.source=USER_INFO basic.auth.user.info=: schema.registry.url=https:// replication.factor=3 ksql.endpoint= ksql.basic.auth.user.info=: ``` ### Destroy a ccloud-stack 1. To destroy a `ccloud-stack` created in the previous step, call the bash script [ccloud_stack_destroy.sh](https://github.com/confluentinc/examples/tree/latest/ccloud/ccloud-stack/ccloud_stack_destroy.sh) and pass in the client properties file auto-generated in the step above. By default, this deletes all resources, including the Confluent Cloud environment specified by the service account ID in the configuration file. ```bash ./ccloud_stack_destroy.sh stack-configs/java-service-account-.config ``` Any Confluent Cloud example uses real Confluent Cloud resources. After you are done running a Confluent Cloud example, manually verify that all Confluent Cloud resources are destroyed to avoid unexpected charges. ## Advanced Options ### Select Cloud Provider and Region By default, the `ccloud-stack` utility creates resources in the cloud provider `aws` in region `us-west-2`. To create resources in another cloud provider or region other than the default, complete the following steps: 1. View the available cloud providers and regions using the Confluent CLI: ```bash confluent kafka region list ``` 2. Create the `ccloud-stack` and override the parameters `CLUSTER_CLOUD` and `CLUSTER_REGION`, as shown in the following example: ```bash CLUSTER_CLOUD=aws CLUSTER_REGION=us-west-2 ./ccloud_stack_create.sh ``` ### Reuse Existing Environment By default, a new `ccloud-stack` creates a new environment. This means that, by default, `./ccloud_stack_create.sh` creates a new environment and `./ccloud_stack_destroy.sh` deletes the environment specified in the configuration file. However, due to Confluent Cloud [environment limits per organization](https://docs.confluent.io/cloud/current/quotas/index.html#organization), it may be desirable to work within an existing environment. When you create a new stack, to reuse an existing environment, set the parameter `ENVIRONMENT` with an existing environment ID, as shown in the example: ```bash ENVIRONMENT=env-oxv5x ./ccloud_stack_create.sh ``` When you destroy resources that were created by `ccloud-stack`, the default behavior is that the environment specified by the service account ID in the configuration file is deleted. However, there are two additional options. To preserve the environment when destroying all the other resources in the `ccloud-stack`, set the parameter `PRESERVE_ENVIRONMENT=true`, as shown in the following example. If you do not specify `PRESERVE_ENVIRONMENT=true`, then the environment specified by the service account ID in the configuration file is deleted. ```bash PRESERVE_ENVIRONMENT=true ./ccloud_stack_destroy.sh stack-configs/java-service-account-.config ``` To destroy the environment when destroying all the other resources in the `ccloud-stack`, but the service account is not part of the environment name (i.e., multiple `ccloud-stacks` were created in the same environment), set the parameter `ENVIRONMENT_NAME_PREFIX=ccloud-stack-`, as shown in the following example. Note that the service account ID in the environment name is not the same as the service account ID in the config name. If you do not specify the environment name prefix, then the destroy function will not be able to identify the proper environment ID to delete. ```bash ENVIRONMENT_NAME_PREFIX=ccloud-stack- ./ccloud_stack_destroy.sh stack-configs/java-service-account-.config ``` ### Automated Workflows If you don’t want to create and destroy a `ccloud-stack` using the provided bash scripts [ccloud_stack_create.sh](https://github.com/confluentinc/examples/tree/latest/ccloud/ccloud-stack/ccloud_stack_create.sh) and [ccloud_stack_destroy.sh](https://github.com/confluentinc/examples/tree/latest/ccloud/ccloud-stack/ccloud_stack_destroy.sh), you may pull in the [ccloud_library](https://github.com/confluentinc/examples/tree/latest/utils/ccloud_library.sh) and call the functions `ccloud::create_ccloud_stack()` and `ccloud::destroy_ccloud_stack()` directly. 1. Get the [ccloud_library](https://github.com/confluentinc/examples/tree/latest/utils/ccloud_library.sh): ```bash curl -f -sS -o ccloud_library.sh https://raw.githubusercontent.com/confluentinc/examples/latest/utils/ccloud_library.sh ``` 2. Source the library ```bash source ./ccloud_library.sh ``` 3. Optionally override the `CLUSTER_CLOUD` and `CLUSTER_REGION` configuration parameters. ```bash CLUSTER_CLOUD=aws CLUSTER_REGION=us-west-2 ``` 4. Run the bash function directly from the command line. To create the `ccloud-stack` without Confluent Cloud ksqlDB: ```bash ccloud::create_ccloud_stack ``` To create the `ccloud-stack` with Confluent Cloud ksqlDB: ```bash ccloud::create_ccloud_stack true ``` 5. To destroy the `ccloud-stack`, run the following command. By default, it deletes all resources, including the Confluent Cloud environment specified by the service account ID in the configuration file. ```bash ccloud::destroy_ccloud_stack $SERVICE_ACCOUNT_ID ``` ## Additional Resources - For a practical guide to configuring, monitoring, and optimizing your Kafka client applications when using Confluent Cloud, see [Developing Client Applications on Confluent Cloud](https://docs.confluent.io/cloud/current/client-apps/index.html). - Read this blog post about [using Confluent Cloud to manage data pipelines that use both on-premise and cloud deployments](https://www.confluent.io/blog/multi-cloud-integration-across-distributed-systems-with-kafka-connect/). - For sample usage of `ccloud-stack`, see [Confluent Cloud Tutorials](/cloud/current/get-started/cloud-demos.html) or [Observability for Apache Kafka Clients to Confluent Cloud demo](../../monitoring/ccloud-observability.md#ccloud-observability-index).