Add Capacity to a Multi-Tenant Physical Kafka Cluster

Add brokers to a multi-tenant Physical Kafka Cluster (PKC) in Confluent Private Cloud when its Logical Kafka Clusters (LKCs) need more capacity than the PKC can deliver. Platform operators perform this procedure.

To add a broker, complete the following steps:

Determine how many brokers to add.
Update the TLS certificate so that it covers the LKC hostnames of the new brokers.
Increase the broker count on the Kafka custom resource (CR).
Add DNS records that resolve the LKC hostnames of the new brokers.
Optional: Verify the in-cluster names for each LKC, if you set createTenantNameServices to true.
Monitor the rebalance that moves partition leaders onto the new brokers. A new broker serves an LKC only after partition leaders for that LKC move to it.

Prerequisites

Before you begin, ensure you have the following:

A multi-tenant PKC that uses host-based external access, such as a load balancer or a route. For more information, see the Deploy a Multi-Tenant Physical Kafka Cluster page.
At least three brokers. Multi-Tenancy stores LKC metadata in an internal topic that requires three replicas.
Permissions to scale the PKC’s Kafka CR, update the certificate Secret, list Service objects, read LogicalKafkaCluster CRs, create DNS records for the PKC’s domain, and reissue the external listener’s certificate.

Step 1: Determine how many brokers to add

Add enough brokers so that the combined usage of all LKCs stays within the PKC’s capacity-planning baseline.

Review each LKC’s current usage and quota configuration. For instructions, see the Monitor and Manage Per-LKC Resource Quotas page.
Compare the combined usage with the PKC’s capacity-planning baseline. For more information about the capacity-planning baseline, see the Deploy a Multi-Tenant Physical Kafka Cluster page.
Calculate how many brokers to add by dividing the capacity shortfall by the per-broker baseline and rounding up. Every broker in a Kafka CR uses the same pod template, so each new broker adds the same baseline capacity as an existing broker. A new broker adds capacity for an LKC only if that LKC has enough partitions for some of its partition leaders to move to the new broker.

Note

The ingressByteRate and egressByteRate values in an LKC’s spec.quotas apply to the whole PKC, not to each broker. Each broker enforces a share of the value. The share starts as an equal split among the brokers that lead at least one of the LKC’s partitions. Confluent for Kubernetes (CFK) also enables dynamic quotas, which adjust each broker’s share to match where the LKC’s traffic goes. No broker’s share can exceed the broker-level maximum.

For more information about moving partition leaders to new brokers, see Step 6.

Step 2: Update the TLS certificate

Reissue the PKC’s TLS certificate before you scale so that it covers the hostnames of each new broker. The new brokers then start with a certificate that already covers them.

CFK assigns ordinals to new brokers sequentially, starting after the highest existing broker ordinal. For example, scaling from three to five brokers creates brokers 3 and 4.

Clients reach a broker through a hostname in the following format:

<lkc-id>-kafka.b<broker-ordinal>.<domain>

In this format, <lkc-id> is the LKC’s ID, which includes the lkc- prefix. The prefix b is the default broker prefix. If you set brokerPrefix in the Kafka CR, such as in spec.listeners.external.externalAccess.loadBalancer.brokerPrefix, use that value instead.

Update the Subject Alternative Name (SAN) entries in the certificate for the TLS approach you chose:

Wildcard SAN: Add a wildcard SAN for each new broker, such as *.b3.mt.example.com. A wildcard matches only one DNS label, so *.mt.example.com doesn’t cover these hostnames.
Per-LKC SAN: For each new broker, add a SAN entry for every LKC on the PKC, such as lkc-x7f2q9-kafka.b3.mt.example.com.

Reissue the certificate, and then update the certificate Secret that the Kafka CR’s external listener references. CFK watches secrets in .pem format and, when the content changes, rolls the existing brokers with under-replicated partition checks.

Wait for the roll to finish before you continue. To check progress, run the following command until the READY column equals REPLICAS:

kubectl get kafka <pkc-name> -n <pkc-namespace>

For more information about the TLS approaches, see the Configure TLS Certificates and DNS for Multi-Tenant PKCs page.

If CFK generates the certificate (autoGeneratedCerts: true) and managedCerts is enabled in the CFK Helm values, update the platform.confluent.io/managed-cert-add-sans annotation on the Kafka CR instead. The annotation holds the complete list of custom SANs, so keep the existing entries and add the wildcard for each new broker, such as *.b3.mt.example.com. CFK doesn’t apply a changed annotation to the existing certificate on its own. To make CFK reissue the certificate with the current SANs, set the platform.confluent.io/force-managed-cert-renewal annotation to true:

kubectl annotate kafka <pkc-name> -n <pkc-namespace> \
  platform.confluent.io/force-managed-cert-renewal="true" --overwrite

The new brokers start with the reissued certificate. The existing brokers keep serving only their own hostnames, which the previous certificate already covers, so you don’t need to restart them.

Step 3: Increase the broker count

Increase the broker count on the PKC’s Kafka CR.

Scale the Kafka CR:
```
kubectl scale kafka <pkc-name> --replicas=<new-broker-count> -n <pkc-namespace>
```
Replace the placeholders with your values:
- <pkc-name>: The name of the PKC’s Kafka CR.
- <new-broker-count>: The total number of brokers you want after scaling, not the number to add.
- <pkc-namespace>: The namespace of the PKC’s Kafka CR.
You don’t need to wait for the new brokers to become ready before you add their DNS records. Each new broker’s load balancer Service gets an external address before the broker is ready.
Confirm that all brokers are ready:
```
kubectl get kafka <pkc-name> -n <pkc-namespace>
```
When the REPLICAS and READY columns show the same number and the STATUS column shows RUNNING, the new brokers are ready.

Step 4: Add DNS records for the new brokers

For each new broker, create one wildcard DNS record, such as *.b3.mt.example.com, that resolves the hostname of every LKC routed to that broker, for example, lkc-x7f2q9-kafka.b3.mt.example.com. Point the record to the new broker’s external IP address or load balancer. You don’t need a record for the broker’s base hostname, and you don’t need to change the bootstrap wildcard record (*.mt.example.com), which already resolves every LKC’s bootstrap hostname.

To find a new broker’s external address, list the PKC’s Service objects and read the EXTERNAL-IP column of the new broker’s load balancer Service:

kubectl get service -n <pkc-namespace>

For example, to add the broker with ordinal 3 (hostname label b3) in the mt.example.com domain, create the following record:

DNS name	Target
`*.b3.mt.example.com`	Broker 3 load balancer

For more information about DNS records for multi-tenant PKCs, see the Configure TLS Certificates and DNS for Multi-Tenant PKCs page.

Until you create a broker’s wildcard record, the bootstrap wildcard record (*.mt.example.com) also matches the new broker’s hostnames. They then resolve to the bootstrap load balancer instead of the new broker, so a successful DNS lookup doesn’t confirm that the new record is in place.

Caution

Create the DNS records as soon as the new brokers’ load balancer Service objects have an external address. By default, Self-Balancing Clusters start a rebalance when brokers are added, as Step 6 describes. Until a new broker’s hostnames resolve to that broker and are covered by the certificate, clients that reach a partition led by that broker see errors or time out.

Step 5 (optional): Verify the in-cluster names for each LKC

Applications inside the Kubernetes cluster reach an LKC through in-cluster names. If spec.multiTenant.createTenantNameServices is true on the PKC’s Kafka CR, complete this step. Avoid this setting unless you can’t edit your cluster’s DNS, because the per-broker Service objects use IP addresses from the cluster’s Service IP range. For more information, see Configure TLS Certificates and DNS for Multi-Tenant PKCs.

If the setting is false, skip this step. If you configured the CoreDNS rewrite rules from the TLS options page, no change is needed, because the rules match any broker number. Otherwise, extend your in-cluster name resolution to cover the new brokers.

If the setting is true, CFK creates a Kubernetes Service for each new broker in every LKC. Verify the Service objects by completing the following steps.

List the Service objects for an LKC:
```
kubectl get service -l platform.confluent.io/logical-cluster-id=<lkc-id> -n <pkc-namespace>
```
Replace the placeholders with your values:
- <lkc-id>: The value of the status.logicalClusterID field in the LogicalKafkaCluster CR.
- <pkc-namespace>: The namespace of the PKC’s Kafka CR. CFK creates these Service objects in that namespace, even if the LKC is in a different one.
Confirm that the output lists a Service named <lkc-id>-kafka-b<broker-ordinal> for each new broker, in addition to the Service objects for the existing brokers and the <lkc-id>-kafka bootstrap Service.

CFK reconciles these Service objects periodically, so allow a few minutes. If a Service is still missing, check the CFK logs.
Repeat the previous steps for each LKC on the PKC.

Step 6: Monitor the rebalance onto the new brokers

CFK enables Self-Balancing Clusters by default, including on multi-tenant PKCs. By default, Self-Balancing Clusters start a rebalance when brokers are added and move partitions onto the new brokers. A new broker starts with no data and leads no partitions, so it adds no throughput for any LKC until data moves to it.

Monitor the rebalance until it completes by using the kafka-rebalance-cluster command. For more information, see Monitor the balancer with kafka-rebalance-cluster. For more information about how CFK scales Kafka clusters, see Scale Confluent Platform Clusters and Balance Data using Confluent for Kubernetes.

Verify the scaled PKC

Confirm that the Kafka CR shows all brokers ready after scaling (REPLICAS equals READY):
```
kubectl get kafka <pkc-name> -n <pkc-namespace>
```
If you have client credentials for an LKC, use the endpoint in the LKC’s status.bootstrapEndpoint field to produce to or consume from a topic partition whose leader is on a new broker. A successful connection confirms that the certificate and DNS records are correct.
In Confluent Control Center, open the Quota Overview page and confirm that each LKC’s usage is within its quota.