# Overview of Confluent Cloud Tutorials and Examples Confluent Cloud is a resilient, scalable streaming data service based on Apache Kafka®, delivered as a fully managed service. Use its web interface and local command-line interface to manage cluster resources, Kafka topics, Schema Registry, and other services. This page describes a few resources to help you build and validate your solutions on Confluent Cloud. ## Cost to run examples Confluent Cloud examples that use actual Confluent Cloud resources might be billable. An example might create a new Confluent Cloud environment, Kafka cluster, topics, ACLs, service accounts, or resources that have hourly charges like connectors and ksqlDB applications. To avoid unexpected charges, carefully [evaluate the cost of resources](../../billing/overview.md#cloud-billing) before you start. After you are done running a Confluent Cloud example, destroy all Confluent Cloud resources to avoid accruing hourly charges for services and verify that they have been deleted. ### Confluent Cloud free trial Use your Confluent Cloud free trial credit to offset the cost of running these examples. For details on how the free trial works, see [Confluent Cloud Free Trial](../free-trial.md#free-trial). ## Confluent Cloud tutorials and examples catalog ### Confluent Cloud Quick Start The [Confluent Cloud Quick Start](https://github.com/confluentinc/examples/tree/latest/cp-quickstart/) is an automated version of the [Confluent Platform Quick Start](/platform/current/platform-quickstart.html), but this one runs in Confluent Cloud. ![Datagen connectors producing pageviews and users data to Kafka for ksqlDB, Schema Registry, and Control Center.](images/_get-started/ksqldb-quickstart-architecture.png) ### ccloud-stack utility The [ccloud-stack Utility for Confluent Cloud](/platform/current/tutorials/examples/ccloud/docs/ccloud-stack.html#ccloud-stack) creates a stack of fully managed services in Confluent Cloud with a single command, which you can then use for learning and building other demos. Do not use this in a production environment. For details on what the script creates, see [Create a stack with the ccloud-stack utility](#id1). ### Confluent Developer tutorials and courses Many more tutorials, courses, and videos are available on [Confluent Developer](https://developer.confluent.io/). This site features full code examples using Kafka, Kafka Streams, and ksqlDB to show real use cases. You can run the tutorials locally or some with Confluent Cloud. ### Client code examples For client examples in several programming languages, including producers and consumers using Confluent Cloud, and producers and consumers using Avro with Confluent Schema Registry, see [Code Examples for Kafka](/platform/current/tutorials/examples/clients/docs/clients-all-examples.html#clients-all-examples). ![Supported Kafka client languages, including Java, Python, Go, and .NET.](images/_get-started/clients-all.png) ### Confluent CLI The [Confluent CLI tutorial](https://docs.confluent.io/confluent-cli/current/beginner-cloud.html) is a fully scripted example that shows you how to interact with Confluent Cloud using the Confluent CLI. It steps through the following workflow: - Create a new environment and specify it as the default. - Create a new Kafka cluster and specify it as the default. - Create a user key/secret pair and specify it as the default. - Produce and consume with the Confluent CLI. - Create a service account key/secret pair. - Run a Java producer: before and after ACLs. - Run a Java producer: showcase a prefix ACL. - Run Connect and `kafka-connect-datagen` connector with permissions. - Run a Java consumer: showcase a wildcard ACL. - Delete the API key, service account, Kafka topics, Kafka cluster, environment, and the log files. ### Observability for Kafka clients to Confluent Cloud The [observability for Kafka clients to Confluent Cloud example](/platform/current/tutorials/examples/ccloud-observability/docs/index.html#ccloud-observability-index) showcases which client metrics to monitor across failure scenarios and dashboards. The clients run against Confluent Cloud. The example creates a Confluent Cloud cluster, Java producers and consumers, Prometheus, Grafana, and exporters. You can apply the same principles to other time-series databases, visualization technologies, and non-Java clients, which generally offer similar metrics. ### Cloud ETL The [cloud ETL example](/platform/current/tutorials/examples/cloud-etl/docs/index.html#cloud-etl) showcases a cloud extract, transform, load (ETL) solution using all fully managed services on Confluent Cloud. Using the Confluent CLI, the example: 1. Creates a source connector that reads data from an AWS Kinesis stream into Confluent Cloud. 2. Runs a Confluent Cloud ksqlDB application that processes that data. 3. Writes the output data, using a sink connector, into cloud storage on the provider of your choice: Google Cloud Storage, Amazon S3, or Azure Blob Storage. ![Cloud ETL flow from Kinesis and Postgres through Confluent Cloud and Flink to Google Cloud Storage, Amazon S3, and Azure Blob.](images/topology.png) ### On-premises Kafka to Confluent Cloud The [hybrid cloud example](/platform/current/tutorials/cp-demo/docs/index.html#cp-demo) and playbook showcase a hybrid Kafka deployment: one cluster is self-managed and runs locally, the other is a Confluent Cloud cluster. Replicator copies the on-premises data to Confluent Cloud so that stream processing can happen in the cloud. ![Hybrid deployment replicating on-premises Kafka data to Confluent Cloud, with a Wikipedia source, stream processing, and Elasticsearch and Kibana.](images/_get-started/cp-demo-overview-with-ccloud.jpg) ### Microservices in the cloud The [microservices cloud example](/platform/current/tutorials/examples/microservices-orders/docs/index.html#tutorial-microservices-orders) showcases an order management workflow targeting Confluent Cloud. Microservices are deployed locally on Docker, and they are configured to use a Kafka cluster, ksqlDB, and Confluent Schema Registry in Confluent Cloud. Kafka Connect is also deployed locally on Docker. It runs a SQL source connector to produce to Confluent Cloud, and it runs an Elasticsearch sink connector to consume from Confluent Cloud. ![Microservices enriching and transforming order data in real time between a JDBC source connector, Kafka, and an Elasticsearch sink connector.](images/_get-started/microservices-demo.png) ## Build your own cloud demo ### Create a stack with the ccloud-stack utility The [ccloud-stack Utility for Confluent Cloud](/platform/current/tutorials/examples/ccloud/docs/ccloud-stack.html#ccloud-stack) creates a stack of fully managed services in Confluent Cloud for learning and building other demos. Do not use this in a production environment. The script uses the Confluent CLI to dynamically do the following in Confluent Cloud: - Create a new environment. - Create a new service account. - Create a new Kafka cluster and associated credentials. - Enable Confluent Cloud Schema Registry and associated credentials. - Create a new ksqlDB application and associated credentials. - Create ACLs with wildcard for the service account. - Generate a local configuration file with all preceding connection information, useful for other demos and automation. ### Auto-generate configurations to connect to Confluent Cloud The configuration generation script reads a configuration file and auto-generates delta configurations for all Confluent Platform components and clients. Use these per-component configurations for Confluent Platform components and clients connecting to Confluent Cloud: * Confluent Platform components: * Schema Registry * ksqlDB data generator * ksqlDB * Confluent Replicator * Confluent Control Center (Legacy) * Kafka Connect * Kafka connector * Kafka command line tools * Kafka clients: * Java (producer/consumer) * Java (Streams) * Python * .NET * Go * Node.js * C++ * OS: * `env` file ### Connect self-managed components to Confluent Cloud This [Docker-based environment](https://github.com/confluentinc/cp-all-in-one/tree/latest/cp-all-in-one-cloud) can be used with Confluent Cloud. The `docker-compose.yml` launches all services in Confluent Platform (except for the Kafka brokers), runs them in containers on localhost, and automatically configures them to connect to Confluent Cloud. Using this as a foundation, you can then add any connectors or applications. ![Confluent Platform components running locally and connecting to Confluent Cloud.](images/_get-started/cp-all-in-one-cloud.png) ### Combine self-managed and fully managed Confluent Cloud components You can chain the ccloud-stack utility and the configuration generation script to build your own hybrid examples that span on-premises and Confluent Cloud, where some self-managed components run on-premises and fully managed services run in Confluent Cloud. For example, you might want an easy way to run a connector not yet available in Confluent Cloud. In this case, you can run a self-managed connect worker and connector on-premises and connect it to your Confluent Cloud cluster. Or perhaps you want to build a Kafka demo in Confluent Cloud and run the REST Proxy client or Confluent Control Center (Legacy) against it. You can build any example with a mix of fully managed services in Confluent Cloud and self-managed components on localhost, in a few easy steps. 1. Create a [ccloud-stack](/platform/current/tutorials/examples/ccloud/docs/ccloud-stack.html#ccloud-stack) of fully managed services in Confluent Cloud. One of the outputs is a local configuration file with key-value pairs of the required connection values to Confluent Cloud. (If you already have provisioned your Confluent Cloud resources, you can skip this step). ```bash ./ccloud_stack_create.sh ``` 2. Run the configuration generation script , passing in that local configuration file created in the previous step as input. This script generates delta configuration files for all Confluent Platform components and clients, including information for bootstrap servers, endpoints, and credentials required to connect to Confluent Cloud. ```bash # stack-configs/java-service-account-.config is generated by step above ./ccloud-generate-cp-configs.sh stack-configs/java-service-account-.config ``` One of the generated delta configuration files from this step is for environment variables, and it resembles [this example](https://github.com/confluentinc/examples/tree/latest/ccloud/template_delta_configs/env.delta), with credentials filled in. ```none export BOOTSTRAP_SERVERS="" export SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='' password='';" export SASL_JAAS_CONFIG_PROPERTY_FORMAT="org.apache.kafka.common.security.plain.PlainLoginModule required username='' password='';" export REPLICATOR_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='' password='';" export BASIC_AUTH_CREDENTIALS_SOURCE="USER_INFO" export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=":" export SCHEMA_REGISTRY_URL="https://" export CLOUD_KEY="" export CLOUD_SECRET="" export KSQLDB_ENDPOINT="" export KSQLDB_BASIC_AUTH_USER_INFO="" ``` 3. Source the preceding delta env file to export variables into the shell environment. ```bash # delta_configs/env.delta is generated by step above source delta_configs/env.delta ``` 4. Run the desired Confluent Platform services locally using [this Docker-based example](https://github.com/confluentinc/cp-all-in-one/tree/latest/cp-all-in-one-cloud). The Docker Compose file launches Confluent Platform services on your localhost and uses environment variable substitution to populate the parameters with the connection values to your Confluent Cloud so that they can connect to Confluent Cloud. If you want to run a single service, you can bring up that service. ```bash docker-compose up -d ``` When running a self-managed connector locally that connects to Confluent Cloud, first add your desired connector to the base Kafka Connect Docker image as described in [Add Connectors or Software](https://docs.confluent.io/home/connect/extending.html), and then substitute that Docker image in your Docker Compose file. 5. See the [library of bash functions](https://github.com/confluentinc/examples/tree/latest/utils/ccloud_library.sh) to learn how to interact with Confluent Cloud through the Confluent CLI. Any Confluent Cloud example uses real Confluent Cloud resources. After you are done running a Confluent Cloud example, manually verify that all Confluent Cloud resources are destroyed to avoid unexpected charges. ## Related content - [Developing Client Applications on Confluent Cloud](../../client-apps/overview.md#client-overview) - [Confluent Replicator to Confluent Cloud Configurations](copy-data-cloud.md#replicator-to-cloud-configurations) - GitHub: [Application Modernization with Confluent Cloud Demo](https://github.com/confluentinc/demo-application-modernization) - GitHub: [Change Data Capture with Confluent Cloud Demo](https://github.com/confluentinc/demo-change-data-capture) - GitHub: [HappyFeet Store Demo](https://github.com/confluentinc/live-labs/tree/master/DIMT2024) (stream data to cloud databases for real-time apps) - GitHub: [Messaging Integration Demo](https://github.com/confluentinc/pmm/tree/master/messaging-modernization)