Scale Kafka Clusters and Balance Data¶
At a high level, adding brokers to a cluster involves a few key steps:
- Define the configuration for each of the new brokers.
- Provision storage, networking, and compute resources to the brokers.
- Start the brokers with the defined configurations and provisioned resources.
- Reassign partitions across the cluster so that the new brokers share the load and the cluster’s overall performance improves.
To automate the above process, Confluent for Kubernetes leverages Self-Balancing, which is enabled by default with CFK.
If you need to manually enable Self-Balancing, see Enable Self-Balancing for the steps.
To scale up the Kafka cluster:
Increase the number of Kafka replicas in the Kafka CR:
Update Kafka with the new settings:
kubectl apply -f <Kafka CR file>
Ensure that proper DNS records are configured for the new brokers, and ensure that the CFK can resolve the new broker hostname, using a command such as
If you are using
hostsfile instead of a DNS service, update
hostsfile with the new brokers information. For example:
Get the new broker IP addresses:
kubectl get services -n <namespace>
Refer to the existing broker host names with the broker prefix, and derive the hostnames of the new brokers.
Add the new broker hosts to the
/etc/hostsfile, and inject the updated file to the CFK pod as described in Adding entries to Pod /etc/hosts.
With Confluent for Kubernetes, you must scale down Kafka clusters one broker at a time. Scaling down Kafka clusters multiple brokers at a time is not supported.
When Confluent for Kubernetes scales down a Kafka cluster by one broker, it deletes the broker pod and the backing persistentVolume. This deletes any partitions that were stored on that broker.
To scale down a Kafka cluster:
Decrease the number of Kafka brokers by one:
kafka: replicas: 4 # In this example, scaling down from 5
Wait an appropriate time for partitions that were on that broker to be replicated to other existing brokers.
You can check if the partitions were replicated by looking at the Kafka metric,
Total Under Replicated Partitions Across Kafka Brokers, in Confluent Control Center UI or in your monitoring solution. The value should be
Depending on the cluster size and the number of topics, this time could vary.
Repeat the above steps, decreasing the number of Kafka brokers by one, and waiting, until the desired broker count is achieved.
To enable Self-Balancing and rebalance Kafka for any uneven load, pass the settings in
configOverrides in the Kafka CR configuration file as shown below.
kafka: configOverrides: server: - confluent.balancer.enable= true - confluent.balancer.heal.uneven.load.trigger=
- Set confluent.balancer.enable
trueto enable Self-Balancing.
- Set confluent.balancer.heal.uneven.load.trigger
ANY_UNEVEN_LOADto balance the load across the cluster whenever an imbalance is detected. The default is
For a complete list of available settings you can use to control Self-Balancing, see Configuration Options and Commands for Self-Balancing Clusters.