Resize a Dedicated Kafka Cluster in Confluent Cloud¶

This topic shows you how to resize existing Dedicated Apache Kafka® clusters. Dedicated clusters are provisioned and billed in terms of Confluent Unit for Kafka (CKU). Evaluating the number of CKUs your cluster requires helps you to reduce costs or scale to meet your streaming needs.

When you resize a Dedicated Kafka cluster, Confluent rebalances the cluster to help ensure a balanced load across the remaining brokers in the cluster. If you’re reducing the size of the cluster, Confluent removes unused brokers at the end of the rebalance operation.

Important

Once you request a resize to your cluster, you cannot request another update until the initially requested resize has completed. For guidance on how long an expansion or shrink might take and for variables that affect that timing, see Dedicated Kafka cluster provisioning time.

Expand a Dedicated Kafka Cluster¶

This section shows you how to expand an existing Dedicated Kafka cluster. To help you decide whether to expand your cluster, see Monitor cluster load and Interpret a high cluster load value.

Prerequisites:

An existing Dedicated Kafka cluster.
Usage-based billing enabled.

You can expand the number of CKUs in the Confluent Cloud Console. When you expand using the console, you get a preview of the adjusted load of the cluster.

Select your Confluent Cloud cluster.
Click the Cluster Settings, the Capacity tab and on the Adjustable limits pane, select Adjust capacity.
View the current usage on your cluster and determine if you are hitting any usage limits. Use the slider to increase the number of CKUs for the cluster. You will see a preview of the reduced capacity before it is applied. If you are satisfied with the changes, click Apply changes.
Confirm your changes and click Continue. For details on how to shrink a cluster, see Shrink a Dedicated Kafka Cluster.

You can expand the number of CKUs. You use a HTTP PATCH call, specifying the cluster and environment ID in the request URL. The call must include a payload that contains the new number of CKUs. For parameter descriptions and more details, see Update Kafka clusters.

PATCH /cmk/v2/clusters/lkc-00000?environment=env-test1 HTTP/1.1
Host: api.confluent.cloud

{
   "spec": {
         "config": {
               "kind": "Dedicated",
               "cku": 3
         },
         "environment": {
               "id": "env-test1""
         }
   }
}

You can expand the number of CKUs with the kafka cluster update command of the Confluent CLI. Following is an example command that increases the CKUs for the lkc-abc123 cluster. For full details, see confluent kafka cluster update.

confluent kafka cluster update lkc-abc123 --cku 3

See Dedicated provisioning time for guidance on how long expansion may take and variables that affect that timing.

Shrink a Dedicated Kafka Cluster¶

This section shows you how to shrink an existing Dedicated Kafka cluster. You can reduce the capacity of a cluster one CKU at a time to the minimum CKU your cluster supports.

Important

Support for shrink operations by more than one CKU at a time is in Limited Availability. To be considered for access before General Availability, contact your account team.

Considerations for CKU limits on resize requests:

Multi-zone Dedicated Kafka clusters require a minimum capacity of 2 CKUs.
Confluent blocks shrink requests that would violate the per CKU dimension limits for partitions or ingress/egress throughput for the requested cluster size. For more information, see CKU limits per cluster.

Prerequisites:

An existing Dedicated Kafka cluster.
Usage-based billing enabled.

You can shrink the number of CKUs in the Confluent Cloud Console. When you shrink using the console, you get a preview of the adjusted load of the cluster.

Select your Confluent Cloud cluster.
Click the Cluster Settings, the Capacity tab and on the Adjustable limits pane, select Adjust capacity.
View the current usage on your cluster and determine if your cluster load is low enough to support a shrink operation. In general, shrinking a highly loaded cluster will result in application performance issues. For more information, see Monitor cluster load, and Kafka clusters.
Use the slider to reduce the number of CKUs for the cluster. You can reduce the capacity of your cluster to any number of CKU your cluster supports.

You will see a preview of the reduced capacity before it is applied. If you are satisfied with the changes, click Apply changes. For details on how to expand a cluster, see Expand a Dedicated Kafka Cluster.

Note

In rare scenarios, such as when you have just expanded your cluster, you might see the following warning in the Cloud Console. If this occurs, wait 1-2 hours and retry the shrink operation.
Confirm your changes and click Continue.

You can shrink the number of CKUs. You use a HTTP PATCH call, specifying the cluster and environment ID in the request URL. The call must include a payload that contains the new number of CKUs. For parameter descriptions and more details, see Update Kafka clusters.

PATCH /cmk/v2/clusters/lkc-00000?environment=env-test1 HTTP/1.1
Host: api.confluent.cloud

{
   "spec": {
         "config": {
               "kind": "Dedicated",
               "cku": 2
         },
         "environment": {
               "id": "env-test1""
         }
   }
}

You can shrink the number of CKUs with the kafka cluster update command of the Confluent CLI. Following is an example command that reduces the CKUs for the lkc-abc123 cluster. For more details, see confluent kafka cluster update.

confluent kafka cluster update lkc-abc123 --cku 2

Important

If the shrink operation fails, the cluster will expand to its original capacity, and you will be notified when the expansion operation is complete.

See Dedicated provisioning time for guidance on how long shrinking may take and variables that affect that timing.