Scale Kafka Clusters and Balance Data
At a high level, adding brokers to a cluster involves a few key steps:
- Define the configuration for each of the new brokers.
- Provision storage, networking, and compute resources to the brokers.
- Start the brokers with the defined configurations and provisioned resources.
- Reassign partitions across the cluster so that the new brokers share the load
and the cluster’s overall performance improves.
To automate the above process, Confluent for Kubernetes leverages Self-Balancing, which is enabled by
default with CFK.
If you need to manually enable Self-Balancing, see Enable Self-Balancing for the steps.
To scale up the Kafka cluster:
Increase the number of Kafka replicas in the Kafka CR:
Update Kafka with the new settings:
kubectl apply -f <Kafka CR file>
Ensure that proper DNS records are configured for the new brokers, and ensure
that the CFK can resolve the new broker hostname, using a command such as
If you are using
hosts file instead of a DNS service, update
file with the new brokers information. For example:
Get the new broker IP addresses:
kubectl get services -n <namespace>
Refer to the existing broker host names with the broker prefix, and derive
the hostnames of the new brokers.
Add the new broker hosts to the
/etc/hosts file, and inject the
updated file to the CFK pod as described in
Adding entries to Pod /etc/hosts.
With Confluent for Kubernetes, you must scale down Kafka clusters one broker at a time. Scaling
down Kafka clusters multiple brokers at a time is not supported.
When Confluent for Kubernetes scales down a Kafka cluster by one broker, it deletes the broker
pod and the backing persistentVolume. This deletes any partitions that were
stored on that broker.
To scale down a Kafka cluster:
Decrease the number of Kafka brokers by one:
replicas: 4 # In this example, scaling down from 5
Wait an appropriate time for partitions that were on that broker to be
replicated to other existing brokers.
You can check if the partitions were replicated by looking at the Kafka
Total Under Replicated Partitions Across Kafka Brokers, in Confluent Control Center
UI or in your monitoring solution. The value should be
Depending on the cluster size and the number of topics, this time could vary.
Repeat the above steps, decreasing the number of Kafka brokers by one, and waiting,
until the desired broker count is achieved.
To enable Self-Balancing and rebalance Kafka for any uneven load, pass the settings in
configOverrides in the Kafka CR configuration file as shown below.
- confluent.balancer.enable= true
For a complete list of available settings you can use to control Self-Balancing, see
Configuration Options and Commands for Self-Balancing