Manage Kafka Clusters on Confluent Cloud¶
Use this topic to understand and manage clusters in Confluent Cloud, using the Confluent Cloud Console, Confluent CLI, and REST API.
Kafka cluster management concepts¶
Apache Kafka® clusters in Confluent Cloud are different from Kafka clusters in general in that Confluent offers a fully managed service. Use the following concepts to learn more.
- Basics
You choose from different types of clusters for use in development, production, and high-traffic workloads. Confluent Cloud is available from all the major cloud providers in regions around the world and provides access to Kafka cluster and topic configuration settings. For more information, see the following topics:
- Fault Tolerance
- Confluent Cloud provides a centralized global control plane and a distributed collection of servers that operate as a cluster. Clusters can span data centers or cloud provider availability zones (AZs) and provide highly scalable and fault-tolerant systems that support mission-critical applications. For more information, see Resilience in Confluent Cloud.
- Dedicated Clusters
Dedicated clusters in Confluent Cloud support options for private networking, self-managed keys, and client quotas for multi-tenant workloads. You provision and manage Dedicated clusters in terms of Confluent Unit for Kafka (CKU). Use CKUs to expand and shrink your cluster. For more information, see the following topics:
- Resize a Dedicated Kafka Cluster in Confluent Cloud
- Protect Data at Rest Using Self-managed Encryption Keys on Confluent Cloud
- Multi-Tenancy and Client Quotas on Confluent Cloud
For more information about private networking, see Manage Networking on Confluent Cloud.
How to work with Kafka clusters¶
Use this section to create, delete, and update clusters.
Requirements
- Operator or admin access to a Confluent Cloud environment.
- API requests must include an authorization header with an API key and secret. For more information, see Authentication in the Confluent Cloud API reference.
Create Kafka clusters¶
You can create clusters using the Confluent Cloud Console, Confluent CLI, and REST API. For a description of cluster types, service level agreements (SLA), and resource quotas for clusters, see Kafka Cluster Types in Confluent Cloud and Service Quotas for Confluent Cloud.
Considerations:
Cloud provider and region cannot be changed after provisioning a cluster.
For data residency and sovereignty requirements, data you produce to a topic and transmit through Kafka does not move out of the geographic region you select. For more information, see the white paper (PDF) Confluent Cloud Security Controls.
Cluster name (
display_name) is a friendly name with the following requirements:- Use 64 characters or less
- Use whitespace, Unicode letters, numbers, and the following special characters: Period (
.), Comma (,), Ampersand (&), Underscore (_), Plus (+), Bar (|), Open square bracket ([), Close square bracket (]), Slash (/), Dash (-)
Tip
The business use of a cluster is a good choice for cluster name.
Choose a cluster type.
Navigate to the clusters page for your environment. If this is your fist cluster, click Create cluster on my own. Otherwise, click + Add cluster.
Choose Basic or Standard and click Begin configuration.
Select a cloud provider tile, Region, and Uptime SLA and click Continue.
Specify Cluster name.
Review your configuration and click Launch cluster.
To verify or add a payment method, or use a promotional code, click Review payment method.
Navigate to the clusters page for your environment. If this is your first cluster, click Create cluster on my own. Otherwise, click + Add cluster.
Choose Enterprise and click Begin configuration.
Select a cloud provider tile, Region, and Uptime SLA and click Continue.
Select I will configure later and click Continue, or click Configure now to configure a gateway.
If you choose to configure, take one of the following options:
Click Continue if a gateway for the selected region is available.
Enter a gateway name, review your selected provider and region, and then click Submit.
Configure the networking using Terraform, the Confluent Cloud Console, or Confluent CLI. For more information, see Use AWS PrivateLink for Serverless Products on Confluent Cloud or Use Azure Private Link for Serverless Products on Confluent Cloud.
Specify Cluster name.
Review your configuration and click Launch cluster.
To verify or add a payment method, or use a promotional code, click Review payment method.
You must contact Confluent to create Freight clusters. With Freight clusters, you first work with the sales team to determine a maximum eCKU for your cluster before you provision.
Navigate to the clusters page for your environment. If this is your first cluster, click Create cluster on my own. Otherwise, click + Add cluster.
Choose Freight and click Begin configuration.
Select a cloud provider tile, Region, and click Continue.
Select I will configure later and click Continue, or click + Create configuration to configure a gateway.
If you choose to configure, take one of the following options:
Click Continue if a gateway for the selected region is available.
Enter a gateway name, review your selected provider and region, and then click Submit.
Configure the networking using Terraform, the Confluent Cloud Console, or Confluent CLI.
Specify Cluster name.
Review your configuration and click Launch cluster.
To verify or add a payment method, or use a promotional code, click Review payment method.
Considerations:
- The number of Confluent Unit for Kafka (CKU) determines the capacity of Dedicated clusters. You can manage CKUs after provisioning. For more information on CKUs and how to update them, see Fixed limits and recommended guidelines and Update Kafka clusters.
- To review network options, see Manage Networking on Confluent Cloud.
- To encrypt your at-rest Kafka cluster data to ensure only the appropriate entity or user can decrypt it, use the cluster creation steps in Protect Data at Rest Using Self-managed Encryption Keys on Confluent Cloud.
To create a Dedicated cluster
Navigate to the clusters page for your environment. If this is your fist cluster, click Create cluster on my own. Otherwise, click + Add cluster.
In Dedicated, specify a cluster size or accept the default size of one CKU and click Begin configuration.
Multi-zone availability requires two CKUs.
Select a cloud provider tile, Region, and Availability and click Continue.
Select Internet and click Continue, or choose a cloud provider to specify a private network:
AWS private network options:
- Select Transit Gateway, VPC Peering, or PrivateLink.
- Choose Use existing, select the existing network, and click Continue.
- Select VPC Peering.
- Choose Create new and enter Network name.
- Select your zones.
- Enter a CIDR block in CIDR for Confluent Cloud Network.
- Click Continue.
- Select Transit Gateway.
- Choose Create new and enter Network name.
- Select your zones.
- Enter a CIDR block in CIDR for Confluent Cloud Network.
- Click Continue.
- Select PrivateLink.
- Choose Create new and enter Network name.
- Select your zones.
- (Optional) Select Private DNS resolution to resolve your cluster endpoints using a private DNS zone. Otherwise, public DNS resolution is used.
- Click Continue.
Azure private network options:
- Select Private Link or VNet Peering.
- Choose Use existing, select the existing network, and click Continue.
- Choose Private Link.
- Choose Create new and enter Network name.
- (Optional) Select Private DNS resolution to resolve your cluster endpoints using a private DNS zone. Otherwise, public DNS resolution is used.
- Click Continue.
- Choose VNet Peering.
- Choose Create new and enter Network name.
- Enter a CIDR block in CIDR for Confluent Cloud Network.
- Click Continue.
GCP private network options:
- Select VPC Peering, or Private Service Connect.
- Choose Use existing, select the existing network, and click Continue.
- Select VPC Peering.
- Choose Create new and enter Network name.
- Select your zones.
- Enter a CIDR block in CIDR for Confluent Cloud Network.
- Click Continue.
- Select Private Service Connect.
- Choose Create new and enter Network name.
- Select your zones.
- (Optional) Select Private DNS resolution to resolve your cluster endpoints using a private DNS zone. Otherwise, public DNS resolution is used.
- Click Continue.
For more information, see Manage Networking on Confluent Cloud.
Specify Automatic security and click Continue.
For self-managed options, follow the on-screen steps to register your key. For more information, see Protect Data at Rest Using Self-managed Encryption Keys on Confluent Cloud.
Specify Cluster name.
Review your configuration and click Launch cluster.
To verify or add a payment method, or use a promotional code, click Review payment method.
Create a cluster.
confluent kafka cluster create <name> [flags]
For example:
confluent kafka cluster create my_new_cluster --cloud "aws" --region "us-west-2"
It may take up to 5 minutes for the Kafka cluster to be ready.
+----------------------+---------------------------------------------------------+
| Current | false |
| ID | lkc-123exa |
| Name | my_new_cluster |
| Type | BASIC |
| Ingress Limit (MB/s) | 250 |
| Egress Limit (MB/s) | 750 |
| Storage | 5 TB |
| Cloud | aws |
| Region | us-west-2 |
| Availability | single-zone |
| Status | PROVISIONING |
| Endpoint | SASL_SSL://pkc-exa45.us-west-2.aws.confluent.cloud:9092 |
| REST Endpoint | https://pkc-exa45.us-west-2.aws.confluent.cloud:443 |
+----------------------+---------------------------------------------------------+
To create a cluster, make a POST request with a JSON payload that specifies the requirements.
POST /cmk/v2/clusters
Request (all cluster types)
When you make the request, include a JSON payload that contains a spec object with the following:
| Parameter | Required or Optional | Description |
|---|---|---|
display_name |
Required | The display name of the cluster. |
availability |
Required |
Availability changes for Basic/Standard/Enterprise clusters Beginning 4/16/2024, the Legacy availability for Basic, Standard and Enterprise clustersThe cluster availability: ``SINGLE_ZONE`` or ``MULTI_ZONE``. |
cloud |
Required | GCP, AZURE or AWS |
region |
Required | A valid region for the cloud provider. |
config |
Required | Object that contains the cluster type: kind: Basic, Standard, Enterprise,
Dedicated, or Freight.
For Dedicated clusters, you must also specify a cku integer value, and
you can specify an optional encryption_key for BYOK clusters.
Note that clusters can be upgraded from Basic to Standard, but cannot be
downgraded from Standard to Basic. |
environment |
Required | Object that contains the environment identifier: id: env-abc123. |
network |
Required for private network | Object that contains the network identifier: id: n-12345 and related environment name. |
If the Dedicated cluster should be created in a private network, create the network first, then specify the network when you create the cluster.
The following example demonstrates the request format for Basic and Standard clusters:
POST /cmk/v2/clusters HTTP/1.1
Host: api.confluent.cloud
{
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"Low",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Basic"
},
"environment":{
"id":"env-a12b34"
}
}
}
The following example demonstrates the request format for an Enterprise cluster in a private network.
POST /cmk/v2/clusters HTTP/1.1
Host: api.confluent.cloud
{
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"High",
"cloud":"AWS",
"region":"us-east-1",
"config":{
"kind":"Enterprise"
},
"environment":{
"id":"env-a12b34"
},
"network":{
"id":"n-12345",
"environment": "env-a12b34"
}
}
}
The following examples demonstrate the request format for Dedicated clusters.
The example below demonstrates the request on a secure public endpoint. Note that you
specify the number of CKUs in the config element:
POST /cmk/v2/clusters HTTP/1.1
Host: api.confluent.cloud
{
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 2
},
"environment":{
"id":"env-a12b34"
}
}
}
The following example demonstrates the request format for a private network. Note that you
specify the network in which to create the cluster in the network element.
POST /cmk/v2/clusters HTTP/1.1
Host: api.confluent.cloud
{
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 2
},
"environment":{
"id":"env-a12b34"
},
"network":{
"id":"n-12345",
"environment": "env-a12b34"
}
}
}
Response
Successful calls return HTTP 202 ACCEPTED with a JSON payload that describes the cluster.
Responses include the following information:
- The cloud provider (AWS, GCP, AZURE) and region for the cluster.
- The cluster status (PROVISIONED, PROVISIONING, FAILED) and
kind(Basic, Standard, Enterprise, Dedicated, and Freight). - Information about the environment that contains the cluster.
- Dedicated clusters only - the number of CKUs allocated to the cluster.
Example response for Basic and Standard clusters:
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
{
"api_version": "cmk/v2",
"id": "abc-f3a90de",
"kind": "Cluster",
"metadata": {
"created_at": "2022-04-22T20:45:26.657894Z",
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"updated_at": "2022-04-22T20:45:26.659364Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "Low",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Basic"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind":"Environment"
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status": {
"phase": "PROVISIONING"
}
}
Example response for Enterprise clusters in a private network
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2023-06-22T20:45:26.657894Z",
"updated_at": "2023-06-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "High",
"cloud": "AWS",
"region": "us-east-1",
"config": {
"kind": "Enterprise"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east-1.aws.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east-1.aws.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind":"Environment"
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
},
"network": {
"id": "n-00000",
"environment": "env-a12b34",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/network=n-00000",
"api_version": "networking/v1",
"kind": "Network"
}
},
"status": {
"phase": "PROVISIONING"
}
}
Example response on a secure public endpoint:
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Dedicated",
"cku": 2
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind":"Environment"
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status": {
"phase": "PROVISIONING",
"cku": 2
}
}
Example response in a private network:
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Dedicated",
"cku": 2
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind":"Environment"
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
},
"network": {
"id": "n-00000",
"environment": "env-a12b34",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/network=n-00000",
"api_version": "networking/v1",
"kind": "Network"
}
},
"status": {
"phase": "PROVISIONING",
"cku": 2
}
}
View Kafka clusters¶
You can view details of a single cluster and lists multiple clusters using the Confluent Cloud Console, Confluent CLI, and REST API.
View a list of Kafka clusters¶
Use this procedure to list clusters.
To list clusters
From the Administration menu, select Environments.
Select an environment.
A list of clusters appears.
List clusters.
confluent kafka cluster list [flags]
For example:
confluent kafka cluster list
Current | ID | Name | Type | Cloud | Region | Availability | Status
----------+------------+--------------------+----------+----------+-----------+--------------+---------
| lkc-123exc | cluster_1 | BASIC | gcp | us-west4 | single-zone | UP
| lkc-123exb | cluster_0 | STANDARD | gcp | us-west4 | single-zone | UP
| lkc-123exa | my_new_cluster | BASIC | aws | us-west-2 | single-zone | UP
To list clusters, make a GET request that specifies the environment.
Request
GET /cmk/v2/clusters?environment={environment_id}
Your request can include the following parameters:
| Parameter | Required or Optional | Description |
|---|---|---|
environment |
Required | Specifies the identifier for the environment that contains the clusters to list. |
page_size |
Optional | Specifies an integer page size for the returned payload. Max value is 100. Default: 10. |
spec.network |
Optional | A network identifier for the clusters to list. Your query can contain multiple network identifiers. |
page_token |
Optional | Contains the opaque pagination token string. |
Response
Successful calls return HTTP 200 OK and a JSON payload that contains the list of clusters for the environment.
Responses include the following information:
- The cloud provider (AWS, GCP, AZURE) and region for the cluster.
- The cluster status (PROVISIONED, PROVISIONING, FAILED) and
kind(Basic, Standard, Enterprise, Dedicated, and Freight). - Information about the environment that contains the cluster.
- Dedicated clusters only - the number of CKUs allocated to the cluster.
Example response for Basic and Standard clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version": "cmk/v2",
"data": [
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "Low",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Basic"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"id": "env-a12b34",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
"api_version": "org/v2",
"kind": "Environment"
}
},
"status": {
"phase": "PROVISIONED"
}
}
],
"kind": "ClusterList",
"metadata": {
"first": "https://api.confluent.cloud/cmk/v2/clusters",
"total_size": 1
}
}
Example response for Enterprise clusters in a private network:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version": "cmk/v2",
"data": [
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "High",
"cloud": "AWS",
"region": "us-east-1",
"config": {
"kind": "Enterprise"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"id": "env-a12b34",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
"api_version": "org/v2",
"kind": "Environment"
},
"network": {
"id": "n-00000",
"environment": "env-a12b34",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34,
"api_version": "networking/v1",
"kind": "Network"
}
},
"status": {
"phase": "PROVISIONED"
}
}
],
"kind": "ClusterList",
"metadata": {
"first": "https://api.confluent.cloud/cmk/v2/clusters",
"total_size": 1
}
}
Example response on a secure public endpoint:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version": "cmk/v2",
"data": [
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Basic"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"id": "env-a12b34",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
"api_version": "org/v2",
"kind": "Environment"
}
},
"status": {
"phase": "PROVISIONED",
"cku": 2
}
}
],
"kind": "ClusterList",
"metadata": {
"first": "https://api.confluent.cloud/cmk/v2/clusters",
"total_size": 1
}
}
Example response in a private network:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version": "cmk/v2",
"data": [
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Basic"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"id": "env-a12b34",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
"api_version": "org/v2",
"kind": "Environment"
},
"network": {
"id": "n-00000",
"environment": "env-a12b34",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34,
"api_version": "networking/v1",
"kind": "Network"
}
},
"status": {
"phase": "PROVISIONED",
"cku": 2
}
}
],
"kind": "ClusterList",
"metadata": {
"first": "https://api.confluent.cloud/cmk/v2/clusters",
"total_size": 1
}
}
View a Kafka cluster¶
Use this procedure to get details about a cluster.
To list a cluster
Navigate to the clusters page for your environment.
Select Cluster Settings to see details about your cluster. To see capacity details, click Capacity.
Describe a cluster.
confluent kafka cluster describe [id] [flags]
For example:
confluent kafka cluster describe lkc-123exa
+----------------------+---------------------------------------------------------+
| Current | true |
| ID | lkc-123exa |
| Name | my_new_cluster |
| Type | BASIC |
| Ingress Limit (MB/s) | 250 |
| Egress Limit (MB/s) | 750 |
| Storage | 5 TB |
| Cloud | aws |
| Region | us-west-2 |
| Availability | single-zone |
| Status | UP |
| Endpoint | SASL_SSL://pkc-exa45.us-west-2.aws.confluent.cloud:9092 |
| REST Endpoint | https://pkc-exa45.us-west-2.aws.confluent.cloud:443 |
+----------------------+---------------------------------------------------------+
To list a cluster, make a GET request that specifies the cluster and environment.
Request (all cluster types)
GET /cmk/v2/clusters/{id}?environment={environment_id}
Your request can include the following parameters:
| Parameter | Required or Optional | Description |
|---|---|---|
id |
Required | Path parameter specifying the identifier for the cluster to read. |
environment |
Required | Query parameter specifying the identifier for the environment that contains the cluster to read. |
Response
Successful calls return HTTP 200 OK and a JSON payload that describes the cluster.
Responses include the following information:
- The cloud provider (AWS, GCP, AZURE) and region for the cluster.
- The cluster status (PROVISIONED, PROVISIONING, FAILED) and kind (Basic, Standard, Enterprise, Dedicated).
- Info about the environment that contains the cluster.
- Dedicated clusters only - the number of CKUs allocated to the cluster.
Example response for Basic and Standard clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at":"2022-04-22T20:45:26.657894Z",
"updated_at":"2022-04-22T20:45:28.045579Z"
},
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"Low",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Standard"
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"api_version": "org/v2",
"id": "env-a12b34",
"kind": "Environment",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED"
}
}
Example response for Enterprise clusters on a private network:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at":"2006-01-02T15:04:05-07:00",
"updated_at":"2006-01-02T15:04:05-07:00"
},
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"High",
"cloud":"AWS",
"region":"us-east-1",
"config":{
"kind":"Enterprise"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east-1.aws.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east-1.aws.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind": "Environment",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
},
"network": {
"id": "n-00000",
"environment": "test",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/network=n-00000",
"api_version": "networking/v1",
"kind": "Network"
}
},
"status":{
"phase":"PROVISIONED"
}
}
Example response on a secure public endpoint:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at":"2006-01-02T15:04:05-07:00",
"updated_at":"2006-01-02T15:04:05-07:00"
},
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 2
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"api_version": "org/v2",
"id": "env-a12b34",
"kind": "Environment",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED",
"cku": 2
}
}
Example response on a private network:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at":"2006-01-02T15:04:05-07:00",
"updated_at":"2006-01-02T15:04:05-07:00"
},
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 2
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind": "Environment",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
},
"network": {
"id": "n-00000",
"environment": "test",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/network=n-00000",
"api_version": "networking/v1",
"kind": "Network"
}
},
"status":{
"phase":"PROVISIONED",
"cku": 2
}
}
Update Kafka clusters¶
You can update a cluster using the Confluent Cloud Console, Confluent CLI, and REST API.
Considerations:
- You can update cluster names for all cluster types.
- For Basic clusters, you can upgrade cluster type to Standard (cluster types cannot be downgraded).
- Upgrades from Basic to Standard must be made on their own, with no other requests, and come with zero downtime.
- For Standard clusters, you can upgrade uptime (SLA downgrades not supported).
- For Dedicated clusters, you can update number of CKUs.
- Shrink operations can reduce the capacity of the cluster to the minimum number of CKU supported by the cluster.
For more information about resizing clusters, see Expand and shrink.
Navigate to the clusters page for your environment.
Select Cluster Settings.
In Identification, click the edit icon.
Enter the new name in Cluster name and click Save changes.
Navigate to the clusters page for your environment.
Select Cluster Settings.
In Cluster details for a Basic cluster, click See details.
In Select Uptime SLA, select an uptime:
- 99.9% Uptime SLA
- 99.99% Uptime SLA
Acknowledge that SLA downgrades are not supported and select Upgrade.
If additional e-CKUs are required, you are prompted. Review New base cost and select Apply changes.
- Navigate to the clusters page for your environment.
- Select Cluster Settings.
- In Cluster details, click See details.
- In Additional E-CKUs required, review New base cost, acknowledge that SLA downgrades are not supported and select Apply changes.
To resize Dedicated clusters with Cloud Console, see Expand a Dedicated Kafka Cluster and Shrink a Dedicated Kafka Cluster.
Update a cluster.
confluent kafka cluster update <id> [flags]
For example:
confluent kafka cluster update lkc-123exa --name "my-new-name"
+----------------------+---------------------------------------------------------+
| Current | true |
| ID | lkc-123exa |
| Name | my-new-name |
| Type | BASIC |
| Ingress Limit (MB/s) | 250 |
| Egress Limit (MB/s) | 750 |
| Storage | 5 TB |
| Cloud | aws |
| Region | us-west-2 |
| Availability | single-zone |
| Status | UP |
| Endpoint | SASL_SSL://pkc-exa45.us-west-2.aws.confluent.cloud:9092 |
| REST Endpoint | https://pkc-exa45.us-west-2.aws.confluent.cloud:443 |
+----------------------+---------------------------------------------------------+
To update the cluster, make a PATCH call to the cluster endpoint.
You specify the environment and cluster identifiers in the request, and include a payload that contains details of the change.
Request (all cluster types)
PATCH /cmk/v2/clusters/{id}
| Parameter | Required or Optional | Description |
|---|---|---|
id |
Required | Path parameter specifying the identifier for the cluster to update. |
The JSON payload must contain a spec object with the following:
environmentobject with anidproperty that identifies the environment
And one or more of the following:
display_nameproperty with a new cluster name valueconfigobject that contains akindproperty specifying the desired cluster type: Standard or Dedicatedconfigobject that contains ackuproperty with an integer value (Dedicated clusters only)
The following example demonstrates the request format for a name change:
PATCH /cmk/v2/clusters/abc-f3a90de HTTP/1.1
Host: api.confluent.cloud
{
"spec": {
"display_name": "updated_name",
"environment": {
"id": "env-a12b34""
}
}
}
The following example demonstrates the request format to change the cluster type:
PATCH /cmk/v2/clusters/abc-f3a90de HTTP/1.1
Host: api.confluent.cloud
{
"spec": {
"config": {
"kind": "Standard"
},
"environment": {
"id": "env-a12b34""
}
}
}
The following example demonstrates the request format to change the number of CKUs:
PATCH /cmk/v2/clusters/abc-f3a90de HTTP/1.1
Host: api.confluent.cloud
{
"spec": {
"config": {
"kind": "Dedicated",
"cku": 3
},
"environment": {
"id": "env-a12b34""
}
}
}
Response
Success returns a 200 OK and JSON payload that describes the updated cluster.
Responses include the following information:
- The cloud provider (AWS, GCP, AZURE) and region for the cluster.
- The cluster status (PROVISIONED, PROVISIONING, FAILED) and
kind(Basic, Standard, Enterprise, Dedicated, and Freight). - Information about the environment that contains the cluster.
- Dedicated clusters only - the number of CKUs allocated to the cluster.
Example response for Basic, Standard, Enterprise, and Freight clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec":{
"display_name":"updated_name",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Basic"
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"id":"env-a12b34",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED"
}
}
Example response for Standard clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec":{
"display_name":"updated_name",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Standard"
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"id":"env-a12b34",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED"
}
}
Example response for Dedicated clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec":{
"display_name":"updated_name",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 3
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"id":"env-a12b34",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED"
"cku": 3
}
}
Delete Kafka clusters¶
You can delete a cluster using the Confluent Cloud Console, Confluent CLI, and REST API. You cannot restore a deleted cluster.
To delete a cluster
Navigate to the clusters page for your environment.
Select Cluster Settings.
Click Delete cluster, enter the cluster name, and click Continue.
Delete a cluster.
confluent kafka cluster delete <id> [flags]
For example (text you must enter is highlighted):
confluent kafka cluster delete lkc-123exa
Are you sure you want to delete Kafka cluster "lkc-123exa"?
To confirm, type "my-new-name". To cancel, press Ctrl-C:
my-new-name
Deleted Kafka cluster "lkc-123exa".
To delete a cluster, make a DELETE request specifying the environment and cluster.
Request (all cluster types)
DELETE /cmk/v2/clusters/{id}?environment={environment_id}
| Parameter | Required or Optional | Description |
|---|---|---|
id |
Required | Path parameter specifying the identifier for the cluster to delete. |
environment |
Required | Query parameter specifying the identifier for the environment that contains the cluster to delete. |
Response
Success returns an HTTP 204 No Content.
Confluent retains deleted clusters for a brief period after you delete them. For more information, contact Confluent Cloud Support.