Create a Kafka Cluster from a Template Using Terraform

What is Terraform?

HashiCorp Terraform is an open source infrastructure-as-code tool that lets you build, change, and version your cloud data infrastructure in a safe, efficient way. You program Terraform with human-readable configuration files that you can version, reuse, share, and deploy in your CI/CD pipelines.

Why Terraform and Kafka?

Use the Confluent Terraform provider to deploy and manage Confluent Cloud infrastructure. The Confluent Terraform provider automates the workflow for managing environments, Apache Kafka® clusters, Kafka topics, and other resources in Confluent Cloud.

These are some of the benefits you get with the Confluent Terraform provider:

  • Human Readable Configuration: Define infrastructure resources declaratively in human-readable configuration files.
  • Manage Critical Confluent Cloud Resources: Manage API keys, environments, Kafka clusters, topics, ACLs, RBAC, Private Networking, and more.
  • Consistent Deployability: Provision and manage your infrastructure safely and efficiently throughout its lifecycle. Package reusable modules to provision multi-cloud resources.
  • Multi-Cloud With Ease: Deploy Confluent Cloud seamlessly across cloud providers.
  • Scale Quickly: Provision complex and dependent infrastructure quickly.
  • Industry Standard: Enable industry standard GitOps and infrastructure-as-code practices.

Using the Confluent Terraform provider

The following example shows a simple Terraform configuration file that provisions a Confluent Cloud environment, a basic Kafka cluster, and a service account.

# Configure the Confluent Provider
terraform {
  required_providers {
    confluent = {
      source  = "confluentinc/confluent"
      version = "1.71.0"
    }
  }
}

provider "confluent" {
  cloud_api_key    = var.confluent_cloud_api_key    # optionally use CONFLUENT_CLOUD_API_KEY env var
  cloud_api_secret = var.confluent_cloud_api_secret # optionally use CONFLUENT_CLOUD_API_SECRET env var
}

resource "confluent_environment" "development" {
  display_name = "Development"

  lifecycle {
    prevent_destroy = true
  }
}

resource "confluent_kafka_cluster" "basic" {
  display_name = "basic_kafka_cluster"
  availability = "SINGLE_ZONE"
  cloud        = "AWS"
  region       = "us-east-2"
  basic {}

  environment {
    id = confluent_environment.development.id
  }

  lifecycle {
    prevent_destroy = true
  }
}

resource "confluent_service_account" "app-manager" {
  display_name = "app-manager"
  description  = "Service account to manage Kafka cluster"
}

# Create more resources ...

You must provide appropriate Confluent Cloud and Kafka cluster credentials to use the provider.

For more information about the resources that you can interact with, see the Confluent Terraform Provider documentation in the Terraform registry.

Sample project

For realistic examples that create multiple Confluent Cloud resources, see the Sample Project for Confluent Terraform Provider. The sample project shows configurations that provision Confluent Cloud resources like these:

  • Basic, Standard, Enterprise and Dedicated Kafka clusters
  • PrivateLink connections
  • RBAC and ACLs for access control
  • Source and sink connectors
  • VPC Peering connections

Source code

Resource Importer

The Resource Importer enables importing your existing Confluent Cloud resources to Terraform Configuration (main.tf) and Terraform State (terraform.tfstate) files.

Resources you can manage

You can provision the following Confluent Cloud resources and get data from these data sources in your Terraform configuration files.

Confluent

Resource Data Source
Confluent Invitation Confluent Invitation

ksqlDB

Resource Data Source
ksqlDB Cluster ksqlDB Cluster

Related content