Overview of Confluent Cloud Tutorials and Examples

Confluent Cloud is a resilient, scalable streaming data service based on Apache Kafka®, delivered as a fully managed service. Use its web interface and local command-line interface to manage cluster resources, Kafka topics, Schema Registry, and other services.

This page describes a few resources to help you build and validate your solutions on Confluent Cloud.

Ready to get started?

Sign up for Confluent Cloud, the fully managed cloud-native service for Apache Kafka® and get started for free using the Cloud quick start.
Download Confluent Platform, the self managed, enterprise-grade distribution of Apache Kafka and get started using the Confluent Platform quick start.

Cost to run examples

Important

Confluent Cloud examples that use actual Confluent Cloud resources might be billable. An example might create a new Confluent Cloud environment, Kafka cluster, topics, ACLs, service accounts, or resources that have hourly charges like connectors and ksqlDB applications. To avoid unexpected charges, carefully evaluate the cost of resources before you start. After you are done running a Confluent Cloud example, destroy all Confluent Cloud resources to avoid accruing hourly charges for services and verify that they have been deleted.

Confluent Cloud free trial

Use your Confluent Cloud free trial credit to offset the cost of running these examples. For details on how the free trial works, see Confluent Cloud Free Trial.

Available examples

Confluent Cloud Quick Start

The Confluent Cloud Quick Start is an automated version of the Confluent Platform Quick Start, but this one runs in Confluent Cloud.

Datagen connectors producing pageviews and users data to Kafka for ksqlDB, Schema Registry, and Control Center.

ccloud-stack utility

The ccloud-stack Utility for Confluent Cloud creates a stack of fully managed services in Confluent Cloud with a single command, which you can then use for learning and building other demos. Do not use this in a production environment. For details on what the script creates, see Create a stack with the ccloud-stack utility.

Tutorials

Many more tutorials, courses, and videos are available on Confluent Developer. This site features full code examples using Kafka, Kafka Streams, and ksqlDB to show real use cases. You can run the tutorials locally or some with Confluent Cloud.

Client code examples

For code examples of producers writing to and consumers reading from Confluent Cloud, or producers and consumers using Avro with Confluent Schema Registry, see Code Examples for Kafka. It provides client examples written in multiple programming languages.

Supported Kafka client languages, including Java, Python, Go, and .NET.

Confluent CLI

The Confluent CLI tutorial is a fully scripted example that shows you how to interact with Confluent Cloud using the Confluent CLI. It steps through the following workflow:

Create a new environment and specify it as the default.
Create a new Kafka cluster and specify it as the default.
Create a user key/secret pair and specify it as the default.
Produce and consume with the Confluent CLI.
Create a service account key/secret pair.
Run a Java producer: before and after ACLs.
Run a Java producer: showcase a prefix ACL.
Run Connect and kafka-connect-datagen connector with permissions.
Run a Java consumer: showcase a wildcard ACL.
Delete the API key, service account, Kafka topics, Kafka cluster, environment, and the log files.

Observability for Kafka clients to Confluent Cloud

The observability for Kafka clients to Confluent Cloud example showcases which client metrics to monitor across failure scenarios and dashboards. The clients run against Confluent Cloud. The example creates a Confluent Cloud cluster, Java producers and consumers, Prometheus, Grafana, and exporters. You can apply the same principles to any other time-series database or visualization technology and non-Java clients. They generally offer similar metrics.

Cloud ETL

The cloud ETL example showcases a cloud extract, transform, load (ETL) solution using all fully managed services on Confluent Cloud. Using the Confluent CLI, the example creates a source connector that reads data from an AWS Kinesis stream into Confluent Cloud, then a Confluent Cloud ksqlDB application processes that data, and then a sink connector writes the output data into cloud storage on the provider of your choice: Google Cloud Storage, Amazon S3, or Azure Blob Storage.

On-premises Kafka to Confluent Cloud

The hybrid cloud example and playbook showcase a hybrid Kafka deployment: one cluster is self-managed and runs locally, the other is a Confluent Cloud cluster. Replicator copies the on-premises data to Confluent Cloud so that stream processing can happen in the cloud.

Hybrid deployment replicating on-premises Kafka data to Confluent Cloud, with a Wikipedia source, stream processing, and Elasticsearch and Kibana.

Microservices in the cloud

The microservices cloud example showcases an order management workflow targeting Confluent Cloud. Microservices are deployed locally on Docker, and they are configured to use a Kafka cluster, ksqlDB, and Confluent Schema Registry in Confluent Cloud. Kafka Connect is also deployed locally on Docker, and it runs a SQL source connector to produce to Confluent Cloud and an Elasticsearch sink connector to consume from Confluent Cloud.

Microservices enriching and transforming order data in real time between a JDBC source connector, Kafka, and an Elasticsearch sink connector.

Build your own cloud demo

Create a stack with the ccloud-stack utility

The ccloud-stack Utility for Confluent Cloud creates a stack of fully managed services in Confluent Cloud. Run with a single command, it is a quick way to create fully managed components in Confluent Cloud, which you can then use for learning and building other demos. Do not use this in a production environment. The script uses the Confluent CLI to dynamically do the following in Confluent Cloud:

Create a new environment.
Create a new service account.
Create a new Kafka cluster and associated credentials.
Enable Confluent Cloud Schema Registry and associated credentials.
Create a new ksqlDB application and associated credentials.
Create ACLs with wildcard for the service account.
Generate a local configuration file with all preceding connection information, useful for other demos and automation.

Auto-generate configurations to connect to Confluent Cloud

The configuration generation script <auto-generate-configs> reads a configuration file and auto-generates delta configurations for all Confluent Platform components and clients. Use these per-component configurations for Confluent Platform components and clients connecting to Confluent Cloud:

Confluent Platform components:
- Schema Registry
- ksqlDB data generator
- ksqlDB
- Confluent Replicator
- Confluent Control Center (Legacy)
- Kafka Connect
- Kafka connector
- Kafka command line tools
Kafka clients:
- Java (producer/consumer)
- Java (Streams)
- Python
- .NET
- Go
- Node.js
- C++
OS:
- env file

Connect self-managed components to Confluent Cloud

This Docker-based environment can be used with Confluent Cloud. The docker-compose.yml launches all services in Confluent Platform (except for the Kafka brokers), runs them in containers on localhost, and automatically configures them to connect to Confluent Cloud. Using this as a foundation, you can then add any connectors or applications.

Confluent Platform components running locally and connecting to Confluent Cloud.

Put it all together

You can chain these utilities to build your own hybrid examples that span on-premises and Confluent Cloud, where some self-managed components run on-premises and fully managed services run in Confluent Cloud.

For example, you might want an easy way to run a connector not yet available in Confluent Cloud. In this case, you can run a self-managed connect worker and connector on-premises and connect it to your Confluent Cloud cluster. Or perhaps you want to build a Kafka demo in Confluent Cloud and run the REST Proxy client or Confluent Control Center (Legacy) against it.

You can build any example with a mix of fully managed services in Confluent Cloud and self-managed components on localhost, in a few easy steps.

Create a ccloud-stack of fully managed services in Confluent Cloud. One of the outputs is a local configuration file with key-value pairs of the required connection values to Confluent Cloud. (If you already have provisioned your Confluent Cloud resources, you can skip this step).
```
./ccloud_stack_create.sh
```

Run the configuration generation script <auto-generate-configs>, passing in that local configuration file created in the previous step as input. This script generates delta configuration files for all Confluent Platform components and clients, including information for bootstrap servers, endpoints, and credentials required to connect to Confluent Cloud.

# stack-configs/java-service-account-<SERVICE_ACCOUNT_ID>.config is generated by step above
./ccloud-generate-cp-configs.sh stack-configs/java-service-account-<SERVICE_ACCOUNT_ID>.config

One of the generated delta configuration files from this step is for environment variables, and it resembles this example, with credentials filled in.

export BOOTSTRAP_SERVERS="<CCLOUD_BOOTSTRAP_SERVER>"
export SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='<CCLOUD_API_KEY>' password='<CCLOUD_API_SECRET>';"
export SASL_JAAS_CONFIG_PROPERTY_FORMAT="org.apache.kafka.common.security.plain.PlainLoginModule required username='<CCLOUD_API_KEY>' password='<CCLOUD_API_SECRET>';"
export REPLICATOR_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='<CCLOUD_API_KEY>' password='<CCLOUD_API_SECRET>';"
export BASIC_AUTH_CREDENTIALS_SOURCE="USER_INFO"
export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO="<SCHEMA_REGISTRY_API_KEY>:<SCHEMA_REGISTRY_API_SECRET>"
export SCHEMA_REGISTRY_URL="https://<SCHEMA_REGISTRY_ENDPOINT>"
export CLOUD_KEY="<CCLOUD_API_KEY>"
export CLOUD_SECRET="<CCLOUD_API_SECRET>"
export KSQLDB_ENDPOINT=""
export KSQLDB_BASIC_AUTH_USER_INFO=""

Source the preceding delta env file to export variables into the shell environment.

# delta_configs/env.delta is generated by step above
source delta_configs/env.delta

Run the desired Confluent Platform services locally using this Docker-based example. The Docker Compose file launches Confluent Platform services on your localhost and uses environment variable substitution to populate the parameters with the connection values to your Confluent Cloud so that they can connect to Confluent Cloud. If you want to run a single service, you can bring up that service.
```
docker-compose up -d <service>
```
When running a self-managed connector locally that connects to Confluent Cloud, first add your desired connector to the base Kafka Connect Docker image as described in Add Connectors or Software, and then substitute that Docker image in your Docker Compose file.
See the library of bash functions to learn how to interact with Confluent Cloud through the Confluent CLI.

Any Confluent Cloud example uses real Confluent Cloud resources. After you are done running a Confluent Cloud example, manually verify that all Confluent Cloud resources are destroyed to avoid unexpected charges.