Tools for Confluent Cloud Clusters

Overview

Confluent provides Terraform scripts to aid developers working with Confluent Cloud to provision tools from Confluent Platform automatically, such as Schema Registry, REST Proxy, Kafka Connect, KSQL and Control Center. Once provisioned, these tools are connected to the Confluent Cloud cluster.

These scripts handle creating the underlying infrastructure details, like VPC, Subnets, Firewalls, Storage, Compute, Load Balancers, as well as bootstrapping any processes that are necessary to install and configure the Confluent Platform tools.

Note

These scripts are provided without support and are intended to be a guideline. Report any issues you encounter via the ccloud-tools GitHub repository, and they will be addressed on a best-effort basis.

Requirements

Introduction

Terraform is a tool for developing, changing and versioning infrastructure safely and efficiently using a Infrastructure-as-Code approach. This enables you to have an infrastructure that is immutable and potentially reusable across different Cloud vendors. The ccloud-tools repository leverages Terraform to provide a superior developer experience for anyone building applications with Confluent Cloud. It allows you to get the tools from Confluent Platform in a snap and in your chosen Cloud provider, while taking care of the details about infrastructure provisioning and software bootstrapping.

The following diagram shows an example of what is created by the scripts. Everything related to the infrastructure is managed by the script, including the VPC, subnets, firewall rules, storage, compute and the load balancers that expose the tools. The tools are created in private subnets with no internet access whatsoever. Inbound access is possible only through the public load balancers, which ensures that the tools are secured within the VPC. The link with the Confluent Cloud cluster is done automatically for you. Finally, the script also manages the scale-out process through multiple availability zones.

This means that if you want two or more copies of a given tool, like Schema Registry, it is created across availability zones automatically, ensuring high availability and fault-tolerance within the region for your tools.

../_images/ccloud-tools-diagram.png

The scripts also provides the ability to provision a bastion server, which might be used to perform local troubleshooting in the compute instances. Since the compute instances are created within private subnets, there is no way to access them directly via SSH. The compute instances are created with firewall rules that give SSH access to the compute instances only from the bastion server. By default, the bastion server is not created, so you need to enable it explicitly to use it.

Step 1: Create Kafka Cluster in Confluent Cloud

Important

This step is for Confluent Cloud Professional users only. Confluent Cloud Enterprise users can skip to Step 2: Get Connectivity Details from your Confluent Cloud Cluster.

  1. Login to Confluent Cloud at https://confluent.cloud.

  2. Click Create cluster.

    ../_images/cloud-create-topic.png
  3. Specify a cluster name, choose a cloud provider, and click Continue. Optionally, you can specify read and write throughput, storage, region, and durability.

    ../_images/cloud-pay-launch.png
  4. Confirm your cluster subscription details, payment information, and click Save and launch cluster.

    ../_images/cloud-enter-credit.png

Step 2: Get Connectivity Details from your Confluent Cloud Cluster

In this step, you retrieve the connectivity details from your Confluent Cloud cluster. The actions shown here are usually performed when you want to install the Confluent Cloud CLI and interact with your cluster. However, you don’t need to perform a full install of the Confluent Cloud CLI. We only need the connectivity details that are created as part of the process.

  1. From the Management -> Clusters page, click the ellipses on the right-hand side of your cluster name and click Client config.

    ../_images/cloud-view-details.png
  2. Follow the on-screen Confluent Cloud installation instructions, starting from the section Configure Confluent Cloud CLI. You don’t need to perform any steps from that section, because you’re not installing the Confluent Cloud CLI. Just click Create Kafka Cluster API key & secret to generate a new credential key pair.

    ../_images/cloud-cli-config.png
  3. Take note of the information provided by this page, especifically the Bootstrap Broker List, the API Key and API Secret, and keep it in a safe place. You’ll provide this information later, during the Terraform setup.

Step 3: Clone the Confluent Cloud Tools GitHub Repository

  1. Clone the Confluent Cloud tools repository from GitHub.

    git clone git@github.com:confluentinc/ccloud-tools.git
    

Step 4: Specify the Confluent Cloud Cluster Connectivity Details

In this step, you configure your Confluent Cloud cluster connectivity details, so that once the tools are provisioned, they can connect to the cluster automatically. The information used here was gathered during Step 2: Get Connectivity Details from your Confluent Cloud Cluster.

  1. Navigate to the /terraform/<provider> directory.

    cd terraform/<provider>
    
  2. Rename the file ccloud.auto.tfvars.example to ccloud.auto.tfvars

  3. Open the ccloud.auto.tfvars in a text editor to edit its contents.

    ###########################################
    ############# Confluent Cloud #############
    ###########################################
    
    ccloud_broker_list = "<CCLOUD_BROKER_LIST>"
    ccloud_access_key = "<CCLOUD_ACCESS_KEY>"
    ccloud_secret_key = "<CCLOUD_SECRET_KEY>"
    
  4. Provide the requested information accordingly and save the changes in the file.

Step 5: Specify the Cloud Provider Credentials Details

In this step, you configure your Cloud credentials, so Terraform can connect to your Cloud provider and perform infrastructure provisioning. The steps provided here are high-level steps, because each Cloud provider has its own way to specify credentials.

  1. Navigate to the /terraform/<provider> directory.

    cd terraform/<provider>
    
  2. Rename the file cloud.auto.tfvars.example to cloud.auto.tfvars

  3. Open the cloud.auto.tfvars in a text editor to edit its contents. The example below shows the information that AWS requires for credentials.

    ###########################################
    ################## AWS ####################
    ###########################################
    
    aws_access_key = "<YOUR_AWS_ACCESS_KEY>"
    aws_secret_key = "<YOUR_AWS_SECRET_KEY>"
    aws_region = "us-east-1"
    
  4. Provide the requested information accordingly and save the changes in the file.

Step 6: Run Terraform

In this step, you run Terraform, so it can create the infrastructure and install the Confluent Platform tools on it.

  1. Navigate to the /terraform/<provider> directory.

    cd terraform/<provider>
    
  2. Initialize the folder to download the required plugins. This step is only required once.

    terraform init
    
  3. Perform a dry-run of the install by planning what is going to be created, modified and destroyed.

    terraform plan
    
  4. Apply the plan configuration in the chosen Cloud provider. After running the command below, you will be asked to acknowledge what is about to be performed. Type yes and press enter.

    terraform apply
    
  5. Once the script finishes, it will print the endpoints from the load balancers that fronts each tool.

    Apply complete! Resources: 56 added, 0 changed, 0 destroyed.
    
    Outputs:
    
    control_center_endpoint = http://control-center-00000000.us-east-1.elb.amazonaws.com
    kafka_connect_endpoint = http://kafka-connect-0000000000.us-east-1.elb.amazonaws.com
    ksql_server_endpoint = http://ksql-server-00000000000000.us-east-1.elb.amazonaws.com
    rest_proxy_endpoint = http://rest-proxy-0000000000000000.us-east-1.elb.amazonaws.com
    schema_registry_endpoint = http://schema-registry-000000.us-east-1.elb.amazonaws.com
    
  6. Optional: Destroy the plan configuration from your Cloud provider. After running the following command, you will be asked to acknowledge what is about to be performed. Type yes and press enter.

    terraform destroy
    

Next Steps