Resize a Dedicated Kafka Cluster in Confluent Cloud

Resize a Dedicated Apache Kafka® cluster by changing its Confluent Unit for Kafka (CKU) count to expand or shrink capacity. Choosing the right number of CKUs helps you reduce costs or scale to meet your streaming needs.

When you resize a Dedicated Kafka cluster, Confluent rebalances the cluster to help ensure a balanced load across the remaining brokers in the cluster. If you’re reducing the size of the cluster, Confluent removes unused brokers at the end of the rebalance operation.

Dedicated provisioning time

Confluent Cloud typically provisions clusters in less than two hours. You receive an email when provisioning is complete.

Sometimes, due to cloud-provider-specific constraints, provisioning can take longer. Check the Confluent Cloud status page for any ongoing service disruptions. Contact Confluent Support if provisioning takes longer than 6 hours.

Provisioning time is excluded from the Confluent SLA.

Dedicated resizing time

Resizing a cluster takes about 30 to 60 minutes per CKU. After you request a Kafka cluster resize, you cannot request another change until the original request completes.

When a cluster is under heavy load or has many partitions to move, resizing can take longer than expected. If you anticipate a scheduled event that requires more CKU capacity, proactively scale up the cluster in advance. Doing so accounts for potential delays caused by resource constraints and the rebalancing process.

During a resize operation, your applications might detect leader elections, but performance remains stable. Supported Kafka clients handle these changes gracefully. You receive an email when the resize operation is complete.

Important

After you request a resize to your cluster, you cannot request another update until the initially requested resize is complete. For guidance about how long an expansion or shrink might take and for variables that affect that timing, refer to Dedicated Kafka cluster provisioning time.

Expand a Dedicated Kafka cluster

Expanding a Dedicated Kafka cluster increases its CKU count to add capacity. To decide whether to expand, see Interpret a high cluster load value.

Prerequisites:

You can expand the number of CKUs in the Confluent Cloud Console. When you expand using the console, you get a preview of the adjusted load of the cluster.

  1. Select your Confluent Cloud cluster.

    Confluent Cluster panel

  2. Click Cluster Settings, then the Capacity tab. On the Adjustable limits pane, select Adjust capacity.

    Confluent Cluster capacity

  3. View the current cluster usage and check whether you are reaching any usage limits. Use the slider to increase the number of CKUs for the cluster. You see a preview of the adjusted capacity before it applies. When the changes look correct, click Apply changes.

    Expand Confluent Cluster capacity

  4. Confirm your changes and click Continue. For details on how to shrink a cluster, see Shrink a Dedicated Kafka cluster.

    Confirm cluster expansion

You can expand the number of CKUs. You use an HTTP PATCH call, specifying the cluster and environment ID in the request URL. The call must include a payload that contains the new number of CKUs. For parameter descriptions and more details, see Update Kafka clusters.

PATCH /cmk/v2/clusters/lkc-00000?environment=env-test1 HTTP/1.1
Host: api.confluent.cloud

{
   "spec": {
         "config": {
               "kind": "Dedicated",
               "cku": 3
         },
         "environment": {
               "id": "env-test1""
         }
   }
}

You can expand the number of CKUs with the kafka cluster update command of the Confluent CLI. The following example increases the CKUs for the lkc-abc123 cluster. For full details, see confluent kafka cluster update.

confluent kafka cluster update lkc-abc123 --cku 3

See Dedicated provisioning time for guidance on how long an expansion takes and what factors affect timing.

Shrink a Dedicated Kafka cluster

Shrinking a Dedicated Kafka cluster reduces its CKU count to remove capacity.

Considerations for CKU limits on resize requests:

  • Multi-zone Dedicated Kafka clusters require a minimum capacity of 2 CKUs.

  • Confluent blocks shrink requests that would violate the per CKU dimension limits for partitions or ingress/egress throughput for the requested cluster size. For more information, see CKU limits per cluster.

Prerequisites:

You can shrink the number of CKUs in the Confluent Cloud Console. When you shrink using the console, you get a preview of the adjusted load of the cluster.

  1. Select your Confluent Cloud cluster.

    Confluent Cluster panel

  2. Click Cluster Settings, then the Capacity tab. On the Adjustable limits pane, select Adjust capacity.

    Confluent Cluster capacity

  3. View the current cluster usage to confirm the load is low enough for a shrink. Shrinking a highly loaded cluster can cause performance issues. For more information, see Interpret a high cluster load value and Kafka clusters.

  4. Use the slider to reduce the number of CKUs for the cluster. You can reduce the capacity of your cluster to any number of CKU your cluster supports.

    You see a preview of the reduced capacity before it applies. When the changes look correct, click Apply changes. For details on how to expand a cluster, see Expand a Dedicated Kafka cluster.

    Shrink cluster capacity slider

    Note

    In rare scenarios, such as when you have just expanded your cluster, you might see the following warning in the Cloud Console. If this occurs, wait 1 to 2 hours and retry the shrink operation.

    Warning that cluster metrics are delayed after a recent resize.

  5. Confirm your changes and click Continue.

    Confirm cluster shrink

You can shrink the number of CKUs. You use an HTTP PATCH call, specifying the cluster and environment ID in the request URL. The call must include a payload that contains the new number of CKUs. For parameter descriptions and more details, see Update Kafka clusters.

PATCH /cmk/v2/clusters/lkc-00000?environment=env-test1 HTTP/1.1
Host: api.confluent.cloud

{
   "spec": {
         "config": {
               "kind": "Dedicated",
               "cku": 2
         },
         "environment": {
               "id": "env-test1""
         }
   }
}

You can shrink the number of CKUs with the kafka cluster update command of the Confluent CLI. The following example reduces the CKUs for the lkc-abc123 cluster. For more details, see confluent kafka cluster update.

confluent kafka cluster update lkc-abc123 --cku 2

Important

If the shrink operation fails, the cluster expands to its original capacity, and you receive a notification when the expansion is complete.

See Dedicated provisioning time for guidance on how long a shrink takes and what factors affect timing.