Create a Kafka Cluster on Confluent Cloud from a Template Using Terraform¶
What is Terraform?¶
HashiCorp Terraform is an open source infrastructure-as-code tool that lets you build, change, and version your cloud data infrastructure in a safe, efficient way. You program Terraform with human-readable configuration files that you can version, reuse, share, and deploy in your CI/CD pipelines.
Why Terraform and Kafka?¶
Use the Confluent Terraform provider to deploy and manage Confluent Cloud infrastructure. The Confluent Terraform provider automates the workflow for managing environments, Apache Kafka® clusters, Kafka topics, and other resources in Confluent Cloud.
These are some of the benefits you get with the Confluent Terraform provider:
- Human Readable Configuration: Define infrastructure resources declaratively in human-readable configuration files.
- Manage Critical Confluent Cloud Resources: Manage API keys, environments, Kafka clusters, topics, ACLs, RBAC, Private Networking, and more.
- Consistent Deployability: Provision and manage your infrastructure safely and efficiently throughout its lifecycle. Package reusable modules to provision multi-cloud resources.
- Multi-Cloud With Ease: Deploy Confluent Cloud seamlessly across cloud providers.
- Scale Quickly: Provision complex and dependent infrastructure quickly.
- Industry Standard: Enable industry standard GitOps and infrastructure-as-code practices.
Confluent Terraform tutorial¶
The following tutorials show how to get started with the Confluent Terraform provider:
Using the Confluent Terraform provider¶
The following example shows a simple Terraform configuration file that provisions a Confluent Cloud environment, a basic Kafka cluster, and a service account.
# Configure the Confluent Provider
terraform {
required_providers {
confluent = {
source = "confluentinc/confluent"
version = "2.11.0"
}
}
}
provider "confluent" {
cloud_api_key = var.confluent_cloud_api_key # optionally use CONFLUENT_CLOUD_API_KEY env var
cloud_api_secret = var.confluent_cloud_api_secret # optionally use CONFLUENT_CLOUD_API_SECRET env var
}
resource "confluent_environment" "development" {
display_name = "Development"
lifecycle {
prevent_destroy = true
}
}
resource "confluent_kafka_cluster" "basic" {
display_name = "basic_kafka_cluster"
availability = "SINGLE_ZONE"
cloud = "AWS"
region = "us-east-2"
basic {}
environment {
id = confluent_environment.development.id
}
lifecycle {
prevent_destroy = true
}
}
resource "confluent_service_account" "app-manager" {
display_name = "app-manager"
description = "Service account to manage Kafka cluster"
}
# Create more resources ...
You must provide appropriate Confluent Cloud and Kafka cluster credentials to use the provider.
For more information about the resources that you can interact with, see the Confluent Terraform Provider documentation in the Terraform registry.
Sample project¶
For realistic examples that create multiple Confluent Cloud resources, see the Sample Project for Confluent Terraform Provider. The sample project shows configurations that provision Confluent Cloud resources like these:
- Basic, Standard, Enterprise and Dedicated Kafka clusters
- PrivateLink connections
- RBAC and ACLs for access control
- Source and sink connectors
- VPC Peering connections
Source code¶
- For the provider source code, see the Confluent Terraform Provider repository on GitHub.
- For sample configuration files, see the examples directory in the source repo.
Resource Importer¶
The Resource Importer
enables importing your existing Confluent Cloud resources to Terraform Configuration
(main.tf
) and Terraform State (terraform.tfstate
) files.
Resources you can manage¶
You can provision the following Confluent Cloud resources and get data from these data sources in your Terraform configuration files.
Confluent¶
Resource | Data Source |
---|---|
Confluent Invitation | Confluent Invitation |
Confluent Provider Integration | Confluent Provider Integration |
Flink¶
Resource | Data Source |
---|---|
Flink Artifact | Flink Artifact |
Flink Compute Pool | Flink Compute Pool |
Flink Statement | – |
– | Flink Region |
Kafka cluster¶
Resource | Data Source |
---|---|
Cluster Link | – |
Connector | – |
Dedicated Kafka Cluster Configuration | – |
Kafka Client Quota | Kafka Client Quota |
Kafka Cluster | Kafka Cluster |
Kafka Mirror Topic | – |
Kafka Topic | Kafka Topic |
ksqlDB¶
Resource | Data Source |
---|---|
ksqlDB Cluster | ksqlDB Cluster |
Metadata¶
Resource | Data Source |
---|---|
Business Metadata | Business Metadata |
Business Metadata Binding | Business Metadata Binding |
Tag | Tag |
Tag Binding | Tag Binding |