Manage Kafka Clusters on Confluent Cloud¶
Use this topic to understand and manage clusters in Confluent Cloud, using the Confluent Cloud Console, Confluent CLI, and REST API.
Kafka cluster management concepts¶
Apache Kafka® clusters in Confluent Cloud are different from Kafka clusters in general in that Confluent offers a fully managed service. Use the following concepts to learn more.
- Basics
You choose from different types of clusters for use in development, production, and high-traffic workloads. Confluent Cloud is available from all the major cloud providers in regions around the world and provides access to Kafka cluster and topic configuration settings. For more information, see the following topics:
- Fault Tolerance
- Confluent Cloud provides a centralized global control plane and a distributed collection of servers that operate as a cluster. Clusters can span data centers or cloud provider availability zones (AZs) and provide highly scalable and fault-tolerant systems that support mission-critical applications. For more information, see Resilience in Confluent Cloud.
- Dedicated Clusters
Dedicated clusters in Confluent Cloud support options for private networking, self-managed keys, and client quotas for multi-tenant workloads. You provision and manage Dedicated clusters in terms of Confluent Unit for Kafka (CKU). Use CKUs to expand and shrink your cluster. For more information, see the following topics:
- Resize a Dedicated Kafka Cluster in Confluent Cloud
- Protect Data at Rest Using Self-managed Encryption Keys
- Multi-tenancy and Client Quotas on Confluent Cloud
For more information about private networking, see Networking in Confluent Cloud.
How to work with Kafka clusters¶
Use this section to create, delete, and update clusters.
Requirements
- Operator or admin access to a Confluent Cloud environment.
- API requests must include an authorization header with an API key and secret. For more information, see Authentication in the Confluent Cloud API reference.
Create Kafka clusters¶
You can create clusters using the Confluent Cloud Console, Confluent CLI, and REST API. For a description of cluster types and resource quotas for clusters, see Kafka Cluster Types in Confluent Cloud and Service Quotas for Confluent Cloud.
Considerations:
Cloud provider, region, and availability cannot be changed after provisioning a cluster.
Cluster name (
display_name) is a friendly name with the following requirements:- Use 64 characters or less
- Use whitespace, Unicode letters, numbers, and the following special characters: Period (
.), Comma (,), Ampersand (&), Underscore (_), Plus (+), Bar (|), Open square bracket ([), Close square bracket (]), Slash (/), Dash (-)
Tip
The business use of a cluster is a good choice for cluster name.
Choose a cluster type.
Navigate to the clusters page for your environment. If this is your fist cluster, click Create cluster on my own. Otherwise, click + Add cluster.
Choose a cluster type and click Begin configuration.
Select a cloud provider tile, Region, and Availability and click Continue.
Specify Cluster name.
Review your configuration and click Launch cluster.
To verify or add a payment method, or use a promotional code, click Review payment method.
Considerations:
- The number of Confluent Unit for Kafka (CKU) determines the capacity of Dedicated clusters. You can manage CKUs after provisioning. For more information on CKUs and how to update them, see Limits per CKU and Update Kafka clusters.
- To review network options, see Networking in Confluent Cloud.
- To encrypt your at-rest Kafka cluster data to ensure only the appropriate entity or user can decrypt it, use the cluster creation steps in Protect Data at Rest Using Self-managed Encryption Keys.
To create a Dedicated cluster
Navigate to the clusters page for your environment. If this is your fist cluster, click Create cluster on my own. Otherwise, click + Add cluster.
In Dedicated, specify a cluster size or accept the default size of one CKU and click Begin configuration.
Multi-zone availability requires two CKUs.
Select a cloud provider tile, Region, and Availability and click Continue.
Select Internet and click Continue, or choose a cloud provider to specify a private network:
AWS private network options:
- Select Transit Gateway, VPC Peering, or PrivateLink.
- Choose Use existing, select the existing network, and click Continue.
- Select VPC Peering.
- Choose Create new and enter Network name.
- Select your zones.
- Enter a CIDR block in CIDR for Confluent Cloud Network.
- Click Continue.
- Select Transit Gateway.
- Choose Create new and enter Network name.
- Select your zones.
- Enter a CIDR block in CIDR for Confluent Cloud Network.
- Click Continue.
- Select PrivateLink.
- Choose Create new and enter Network name.
- Select your zones.
- (Optional) Select Private DNS resolution to resolve your cluster endpoints using a private DNS zone. Otherwise, public DNS resolution is used.
- Click Continue.
Azure private network options:
- Select Private Link or VNet Peering.
- Choose Use existing, select the existing network, and click Continue.
- Choose Private Link.
- Choose Create new and enter Network name.
- (Optional) Select Private DNS resolution to resolve your cluster endpoints using a private DNS zone. Otherwise, public DNS resolution is used.
- Click Continue.
- Choose VNet Peering.
- Choose Create new and enter Network name.
- Enter a CIDR block in CIDR for Confluent Cloud Network.
- Click Continue.
GPC private network options:
- Select VPC Peering, or Private Service Connect.
- Choose Use existing, select the existing network, and click Continue.
- Select VPC Peering.
- Choose Create new and enter Network name.
- Select your zones.
- Enter a CIDR block in CIDR for Confluent Cloud Network.
- Click Continue.
- Select Private Service Connect.
- Choose Create new and enter Network name.
- Select your zones.
- (Optional) Select Private DNS resolution to resolve your cluster endpoints using a private DNS zone. Otherwise, public DNS resolution is used.
- Click Continue.
For more information, see Networking in Confluent Cloud.
Specify Automatic security and click Continue.
For self-managed options, follow the on-screen steps to register your key. For more information, see Protect Data at Rest Using Self-managed Encryption Keys.
Specify Cluster name.
Review your configuration and click Launch cluster.
To verify or add a payment method, or use a promotional code, click Review payment method.
Create a cluster.
confluent kafka cluster create <name> [flags]
For example:
confluent kafka cluster create my_new_cluster --cloud "aws" --region "us-west-2"
It may take up to 5 minutes for the Kafka cluster to be ready.
+----------------------+---------------------------------------------------------+
| Current | false |
| ID | lkc-123exa |
| Name | my_new_cluster |
| Type | BASIC |
| Ingress Limit (MB/s) | 250 |
| Egress Limit (MB/s) | 750 |
| Storage | 5 TB |
| Provider | aws |
| Region | us-west-2 |
| Availability | single-zone |
| Status | PROVISIONING |
| Endpoint | SASL_SSL://pkc-exa45.us-west-2.aws.confluent.cloud:9092 |
| REST Endpoint | https://pkc-exa45.us-west-2.aws.confluent.cloud:443 |
+----------------------+---------------------------------------------------------+
To create a cluster, make a POST request with a JSON payload that specifies the requirements.
POST /cmk/v2/clusters
Request (all cluster types)
When you make the request, include a JSON payload that contains a spec object with the following:
| Parameter | Required or Optional | Description |
|---|---|---|
display_name |
Required | The display name of the cluster. |
availability |
Required | The cluster availability: SINGLE_ZONE or MULTI_ZONE. |
cloud |
Required | GCP, AZURE or AWS |
region |
Required | A valid region for the cloud provider. |
config |
Required | Object that contains the cluster type: kind: Basic, Standard, Enterprise, or
Dedicated.
For Dedicated clusters, you must also specify a cku integer value, and
you can specify an optional encryption_key for BYOK clusters.
Note that clusters can be upgraded from Basic to Standard, but cannot be
downgraded from Standard to Basic. |
environment |
Required | Object that contains the environment identifier: id: env-abc123. |
network |
Required for private network | Object that contains the network identifier: id: n-12345 and related environment name. |
If the Dedicated cluster should be created in a private network, create the network first, then specify the network when you create the cluster.
The following example demonstrates the request format for Basic, Standard, and Enterprise clusters:
POST /cmk/v2/clusters HTTP/1.1
Host: api.confluent.cloud
{
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Basic"
},
"environment":{
"id":"env-a12b34"
}
}
}
The following example demonstrates the request format for an Enterprise cluster in a private network.
POST /cmk/v2/clusters HTTP/1.1
Host: api.confluent.cloud
{
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"MULTI_ZONE",
"cloud":"AWS",
"region":"us-east-1",
"config":{
"kind":"Enterprise"
},
"environment":{
"id":"env-a12b34"
},
"network":{
"id":"n-12345",
"environment": "env-a12b34"
}
}
}
The following example demonstrates the request format for Dedicated clusters.
Note that you specify the number of CKUs in the config element:
POST /cmk/v2/clusters HTTP/1.1
Host: api.confluent.cloud
{
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 2
},
"environment":{
"id":"env-a12b34"
}
}
}
The following example demonstrates the request format for a Dedicated cluster in a private network.
Note that you specify the network in which to create the cluster in the network element.
POST /cmk/v2/clusters HTTP/1.1
Host: api.confluent.cloud
{
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 2
},
"environment":{
"id":"env-a12b34"
},
"network":{
"id":"n-12345",
"environment": "env-a12b34"
}
}
}
Response
Successful calls return HTTP 202 ACCEPTED with a JSON payload that describes the cluster.
Responses include the following information:
- The cloud provider (AWS, GCP, AZURE) and region for the cluster.
- The cluster status (PROVISIONED, PROVISIONING, FAILED) and
kind(Basic, Standard, Enterprise and Dedicated). - Information about the environment that contains the cluster.
- Dedicated clusters only - the number of CKUs allocated to the cluster.
Example response for Basic and Standard clusters:
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
{
"api_version": "cmk/v2",
"id": "abc-f3a90de",
"kind": "Cluster",
"metadata": {
"created_at": "2022-04-22T20:45:26.657894Z",
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"updated_at": "2022-04-22T20:45:26.659364Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Basic"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind":"Environment"
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status": {
"phase": "PROVISIONING"
}
}
Example response for Enterprise clusters in a private network
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2023-06-22T20:45:26.657894Z",
"updated_at": "2023-06-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "MULTI_ZONE",
"cloud": "AWS",
"region": "us-east-1",
"config": {
"kind": "Enterprise"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east-1.aws.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east-1.aws.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind":"Environment"
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
},
"network": {
"id": "n-00000",
"environment": "env-a12b34",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/network=n-00000",
"api_version": "networking/v1",
"kind": "Network"
}
},
"status": {
"phase": "PROVISIONING"
}
}
Example response for Dedicated clusters on the public internet:
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Dedicated",
"cku": 2
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind":"Environment"
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status": {
"phase": "PROVISIONING",
"cku": 2
}
}
Example response for Dedicated clusters in a private network
HTTP/1.1 202 ACCEPTED
Content-Type: application/json
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Dedicated",
"cku": 2
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind":"Environment"
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
},
"network": {
"id": "n-00000",
"environment": "env-a12b34",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/network=n-00000",
"api_version": "networking/v1",
"kind": "Network"
}
},
"status": {
"phase": "PROVISIONING",
"cku": 2
}
}
View Kafka clusters¶
You can view details of a single cluster and lists multiple clusters using the Confluent Cloud Console, Confluent CLI, and REST API.
View a list of Kafka clusters¶
Use this procedure to list clusters.
To list clusters
From the Administration menu, select Environments.
Select an environment.
A list of clusters appears.
List clusters.
confluent kafka cluster list [flags]
For example:
confluent kafka cluster list
Current | ID | Name | Type | Provider | Region | Availability | Status
----------+------------+--------------------+----------+----------+-----------+--------------+---------
| lkc-123exc | cluster_1 | BASIC | gcp | us-west4 | single-zone | UP
| lkc-123exb | cluster_0 | STANDARD | gcp | us-west4 | single-zone | UP
| lkc-123exa | my_new_cluster | BASIC | aws | us-west-2 | single-zone | UP
To list clusters, make a GET request that specifies the environment.
Request
GET /cmk/v2/clusters?environment={environment_id}
Your request can include the following parameters:
| Parameter | Required or Optional | Description |
|---|---|---|
environment |
Required | Specifies the identifier for the environment that contains the clusters to list. |
page_size |
Optional | Specifies an integer page size for the returned payload. Max value is 100. Default: 10. |
spec.network |
Optional | A network identifier for the clusters to list. Your query can contain multiple network identifiers. |
page_token |
Optional | Contains the opaque pagination token string. |
Response
Successful calls return HTTP 200 OK and a JSON payload that contains the list of clusters for the environment.
Responses include the following information:
- The cloud provider (AWS, GCP, AZURE) and region for the cluster.
- The cluster status (PROVISIONED, PROVISIONING, FAILED) and
kind(Basic, Standard, Enterprise and Dedicated). - Information about the environment that contains the cluster.
- Dedicated clusters only - the number of CKUs allocated to the cluster.
Example response for Basic and Standard clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version": "cmk/v2",
"data": [
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Basic"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"id": "env-a12b34",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
"api_version": "org/v2",
"kind": "Environment"
}
},
"status": {
"phase": "PROVISIONED"
}
}
],
"kind": "ClusterList",
"metadata": {
"first": "https://api.confluent.cloud/cmk/v2/clusters",
"total_size": 1
}
}
Example response for Enterprise clusters in a private network:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version": "cmk/v2",
"data": [
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "MULTI_ZONE",
"cloud": "AWS",
"region": "us-east-1",
"config": {
"kind": "Enterprise"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"id": "env-a12b34",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
"api_version": "org/v2",
"kind": "Environment"
},
"network": {
"id": "n-00000",
"environment": "env-a12b34",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34,
"api_version": "networking/v1",
"kind": "Network"
}
},
"status": {
"phase": "PROVISIONED"
}
}
],
"kind": "ClusterList",
"metadata": {
"first": "https://api.confluent.cloud/cmk/v2/clusters",
"total_size": 1
}
}
Example response for Dedicated clusters on the public internet:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version": "cmk/v2",
"data": [
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Basic"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"id": "env-a12b34",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
"api_version": "org/v2",
"kind": "Environment"
}
},
"status": {
"phase": "PROVISIONED",
"cku": 2
}
}
],
"kind": "ClusterList",
"metadata": {
"first": "https://api.confluent.cloud/cmk/v2/clusters",
"total_size": 1
}
}
Example response for Dedicated clusters in a private network:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version": "cmk/v2",
"data": [
{
"api_version": "cmk/v2",
"kind": "Cluster",
"id": "abc-f3a90de",
"metadata": {
"self": "https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name": "crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec": {
"display_name": "ProdKafkaCluster",
"availability": "SINGLE_ZONE",
"cloud": "GCP",
"region": "us-east4",
"config": {
"kind": "Basic"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"id": "env-a12b34",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34",
"api_version": "org/v2",
"kind": "Environment"
},
"network": {
"id": "n-00000",
"environment": "env-a12b34",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34,
"api_version": "networking/v1",
"kind": "Network"
}
},
"status": {
"phase": "PROVISIONED",
"cku": 2
}
}
],
"kind": "ClusterList",
"metadata": {
"first": "https://api.confluent.cloud/cmk/v2/clusters",
"total_size": 1
}
}
View a Kafka cluster¶
Use this procedure to get details about a cluster.
To list a cluster
Navigate to the clusters page for your environment.
Select Cluster Settings to see details about your cluster. To see capacity details, click Capacity.
Describe a cluster.
confluent kafka cluster describe [id] [flags]
For example:
confluent kafka cluster describe lkc-123exa
+----------------------+---------------------------------------------------------+
| Current | true |
| ID | lkc-123exa |
| Name | my_new_cluster |
| Type | BASIC |
| Ingress Limit (MB/s) | 250 |
| Egress Limit (MB/s) | 750 |
| Storage | 5 TB |
| Provider | aws |
| Region | us-west-2 |
| Availability | single-zone |
| Status | UP |
| Endpoint | SASL_SSL://pkc-exa45.us-west-2.aws.confluent.cloud:9092 |
| REST Endpoint | https://pkc-exa45.us-west-2.aws.confluent.cloud:443 |
+----------------------+---------------------------------------------------------+
To list a cluster, make a GET request that specifies the cluster and environment.
Request (all cluster types)
GET /cmk/v2/clusters/{id}?environment={environment_id}
Your request can include the following parameters:
| Parameter | Required or Optional | Description |
|---|---|---|
id |
Required | Path parameter specifying the identifier for the cluster to read. |
environment |
Required | Query parameter specifying the identifier for the environment that contains the cluster to read. |
Response
Successful calls return HTTP 200 OK and a JSON payload that describes the cluster.
Responses include the following information:
- The cloud provider (AWS, GCP, AZURE) and region for the cluster.
- The cluster status (PROVISIONED, PROVISIONING, FAILED) and kind (Basic, Standard, Enterprise, Dedicated).
- Info about the environment that contains the cluster.
- Dedicated clusters only - the number of CKUs allocated to the cluster.
Example response for Basic and Standard clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at":"2022-04-22T20:45:26.657894Z",
"updated_at":"2022-04-22T20:45:28.045579Z"
},
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Standard"
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"api_version": "org/v2",
"id": "env-a12b34",
"kind": "Environment",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED"
}
}
Example response for Enterprise clusters on a private network:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at":"2006-01-02T15:04:05-07:00",
"updated_at":"2006-01-02T15:04:05-07:00"
},
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"MULTI_ZONE",
"cloud":"AWS",
"region":"us-east-1",
"config":{
"kind":"Enterprise"
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east-1.aws.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east-1.aws.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind": "Environment",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
},
"network": {
"id": "n-00000",
"environment": "test",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/network=n-00000",
"api_version": "networking/v1",
"kind": "Network"
}
},
"status":{
"phase":"PROVISIONED"
}
}
Example response for Dedicated clusters on the public internet:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at":"2006-01-02T15:04:05-07:00",
"updated_at":"2006-01-02T15:04:05-07:00"
},
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 2
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"api_version": "org/v2",
"id": "env-a12b34",
"kind": "Environment",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED",
"cku": 2
}
}
Example response for Dedicated clusters on a private network:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at":"2006-01-02T15:04:05-07:00",
"updated_at":"2006-01-02T15:04:05-07:00"
},
"spec":{
"display_name":"ProdKafkaCluster",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 2
},
"kafka_bootstrap_endpoint": "abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint": "https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment": {
"api_version": "org/v2",
"id": "env-a12b34",
"kind": "Environment",
"related": "https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name": "crn://confluent.cloud/organization=1234abcd-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
},
"network": {
"id": "n-00000",
"environment": "test",
"related": "https://api.confluent.cloud/networking/v1/networks/n-00000",
"resource_name": "https://api.confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/network=n-00000",
"api_version": "networking/v1",
"kind": "Network"
}
},
"status":{
"phase":"PROVISIONED",
"cku": 2
}
}
Update Kafka clusters¶
You can update a cluster using the Confluent Cloud Console, Confluent CLI, and REST API.
Considerations:
You can update cluster type, cluster names and number of CKUs (Dedicated clusters only).
Upgrades from Basic to Standard must be made on their own, with no other requests.
Clusters cannot be downgraded.
Shrink operations can reduce the capacity of the cluster one (1) CKU at a time to the minimum number of CKU supported by the cluster.
Note
Support for shrink operations by more than one CKU at a time is in Limited Availability. To be considered for access before General Availability, contact your account team.
For more information about resizing clusters, see Expand and shrink.
Navigate to the clusters page for your environment.
Select Cluster Settings.
In Identification, click the edit icon.
Enter the new name in Cluster name and click Save changes.
Navigate to the clusters page for your environment.
Select Cluster Settings.
In Cluster type for a Basic cluster, click the edit icon.
In Standard cluster, click Upgrade, acknowledge that you understand cluster downgrades are not supported, and click Instant upgrade.
To resize Dedicated clusters with Cloud Console, see Expand a Dedicated Kafka Cluster and Shrink a Dedicated Kafka Cluster.
Update a cluster.
confluent kafka cluster update <id> [flags]
For example:
confluent kafka cluster update lkc-123exa --name "my-new-name"
+----------------------+---------------------------------------------------------+
| Current | true |
| ID | lkc-123exa |
| Name | my-new-name |
| Type | BASIC |
| Ingress Limit (MB/s) | 250 |
| Egress Limit (MB/s) | 750 |
| Storage | 5 TB |
| Provider | aws |
| Region | us-west-2 |
| Availability | single-zone |
| Status | UP |
| Endpoint | SASL_SSL://pkc-exa45.us-west-2.aws.confluent.cloud:9092 |
| REST Endpoint | https://pkc-exa45.us-west-2.aws.confluent.cloud:443 |
+----------------------+---------------------------------------------------------+
To update the cluster, make a PATCH call to the cluster endpoint.
You specify the environment and cluster identifiers in the request, and include a payload that contains details of the change.
Request (all cluster types)
PATCH /cmk/v2/clusters/{id}
| Parameter | Required or Optional | Description |
|---|---|---|
id |
Required | Path parameter specifying the identifier for the cluster to update. |
The JSON payload must contain a spec object with the following:
environmentobject with anidproperty that identifies the environment
And one or more of the following:
display_nameproperty with a new cluster name valueconfigobject that contains akindproperty specifying the desired cluster type: Standard or Dedicatedconfigobject that contains ackuproperty with an integer value (Dedicated clusters only)
The following example demonstrates the request format for a name change:
PATCH /cmk/v2/clusters/abc-f3a90de HTTP/1.1
Host: api.confluent.cloud
{
"spec": {
"display_name": "updated_name",
"environment": {
"id": "env-a12b34""
}
}
}
The following example demonstrates the request format to change the cluster type:
PATCH /cmk/v2/clusters/abc-f3a90de HTTP/1.1
Host: api.confluent.cloud
{
"spec": {
"config": {
"kind": "Standard"
},
"environment": {
"id": "env-a12b34""
}
}
}
The following example demonstrates the request format to change the number of CKUs:
PATCH /cmk/v2/clusters/abc-f3a90de HTTP/1.1
Host: api.confluent.cloud
{
"spec": {
"config": {
"kind": "Dedicated",
"cku": 3
},
"environment": {
"id": "env-a12b34""
}
}
}
Response
Success returns a 200 OK and JSON payload that describes the updated cluster.
Responses include the following information:
- The cloud provider (AWS, GCP, AZURE) and region for the cluster.
- The cluster status (PROVISIONED, PROVISIONING, FAILED) and
kind(Basic, Standard, Enterprise and Dedicated). - Information about the environment that contains the cluster.
- Dedicated clusters only - the number of CKUs allocated to the cluster.
Example response for Basic, Standard, and Enterprise clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec":{
"display_name":"updated_name",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Basic"
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"id":"env-a12b34",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED"
}
}
Example response for Standard clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec":{
"display_name":"updated_name",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Standard"
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"id":"env-a12b34",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED"
}
}
Example response for Dedicated clusters:
HTTP/1.1 200 OK
Content-Type: application/json
{
"api_version":"cmk/v2",
"kind":"Cluster",
"id":"abc-f3a90de",
"metadata":{
"self":"https://api.confluent.cloud/v2/kafka-clusters/abc-f3a90de",
"resource_name":"crn://confluent.cloud/kafka=abc-f3a90de",
"created_at": "2022-04-22T20:45:26.657894Z",
"updated_at": "2022-04-22T21:13:55.742641944Z"
},
"spec":{
"display_name":"updated_name",
"availability":"SINGLE_ZONE",
"cloud":"GCP",
"region":"us-east4",
"config":{
"kind":"Dedicated",
"cku": 3
},
"kafka_bootstrap_endpoint":"abc-00000-00000.us-east4.gcp.glb.confluent.cloud:9092",
"http_endpoint":"https://abc-00000-00000.us-east4.gcp.glb.confluent.cloud",
"environment":{
"id":"env-a12b34",
"related":"https://api.confluent.cloud/v2/environments/env-a12b34",
"resource_name":"crn://confluent.cloud/organization=abcd41c4-edef-46ac-8a41-c49e44a3fd9a/environment=env-a12b34"
}
},
"status":{
"phase":"PROVISIONED"
"cku": 3
}
}
Delete Kafka clusters¶
You can delete a cluster using the Confluent Cloud Console, Confluent CLI, and REST API. You cannot restore a deleted cluster.
To delete a cluster
Navigate to the clusters page for your environment.
Select Cluster Settings.
Click Delete cluster, enter the cluster name, and click Continue.
Delete a cluster.
confluent kafka cluster delete <id> [flags]
For example (text you must enter is highlighted):
confluent kafka cluster delete lkc-123exa
Are you sure you want to delete Kafka cluster "lkc-123exa"?
To confirm, type "my-new-name". To cancel, press Ctrl-C:
my-new-name
Deleted Kafka cluster "lkc-123exa".
To delete a cluster, make a DELETE request specifying the environment and cluster.
Request (all cluster types)
DELETE /cmk/v2/clusters/{id}?environment={environment_id}
| Parameter | Required or Optional | Description |
|---|---|---|
id |
Required | Path parameter specifying the identifier for the cluster to delete. |
environment |
Required | Query parameter specifying the identifier for the environment that contains the cluster to delete. |
Response
Success returns an HTTP 204 No Content.
Confluent retains deleted clusters for a brief period after you delete them. For more information, contact Confluent Cloud Support.