<a id="multi-tenancy-scale-a-multi-tenant-pkc"></a>

# Add Capacity to a Multi-Tenant Physical Kafka Cluster

Add brokers to a multi-tenant Physical Kafka Cluster (PKC) in Confluent Private Cloud when
its Logical Kafka Clusters (LKCs) need more capacity than the PKC can
deliver. Platform operators perform this procedure.

To add a broker, complete the following steps:

1. [Determine how many brokers to add](#multi-tenancy-scale-step-1).
2. [Update the TLS certificate](#multi-tenancy-scale-step-2) so that
   it covers the LKC hostnames of the new brokers.
3. [Increase the broker count](#multi-tenancy-scale-step-3) on the
   `Kafka` custom resource (CR).
4. [Add DNS records](#multi-tenancy-scale-step-4) that resolve the
   LKC hostnames of the new brokers.
5. Optional: [Verify the in-cluster names](#multi-tenancy-scale-step-5)
   for each LKC, if you set `createTenantNameServices` to `true`.
6. [Monitor the rebalance](#multi-tenancy-scale-step-6) that moves
   partition leaders onto the new brokers. A new broker serves an LKC
   only after partition leaders for that LKC move to it.

## Prerequisites

Before you begin, ensure you have the following:

- A multi-tenant PKC that uses host-based external access, such as a
  load balancer or a route. For more information, see the Deploy a
  Multi-Tenant Physical Kafka Cluster page.
- At least three brokers. Multi-Tenancy stores LKC metadata in an
  internal topic that requires three replicas.
- Permissions to scale the PKC’s `Kafka` CR, update the certificate
  `Secret`, list `Service` objects, read `LogicalKafkaCluster` CRs,
  create DNS records for the PKC’s domain, and reissue the external
  listener’s certificate.


<a id="multi-tenancy-scale-step-1"></a>

## Step 1: Determine how many brokers to add

Add enough brokers so that the combined usage of all LKCs stays within
the PKC’s capacity-planning baseline.

1. Review each LKC’s current usage and quota configuration. For
   instructions, see the Monitor and Manage Per-LKC Resource Quotas page.
2. Compare the combined usage with the PKC’s capacity-planning
   baseline. For more information about the capacity-planning baseline,
   see the Deploy a Multi-Tenant Physical Kafka Cluster page.
3. Calculate how many brokers to add by dividing the capacity shortfall
   by the per-broker baseline and rounding up. Every broker in a
   `Kafka` CR uses the same pod template, so each new broker adds the
   same baseline capacity as an existing broker. A new broker adds
   capacity for an LKC only if that LKC has enough partitions for some
   of its partition leaders to move to the new broker.


#### NOTE
The `ingressByteRate` and `egressByteRate` values in an LKC’s
`spec.quotas` apply to the whole PKC, not to each broker. Each
broker enforces a share of the value. The share starts as an equal
split among the brokers that lead at least one of the LKC’s
partitions. Confluent for Kubernetes (CFK) also enables dynamic quotas, which adjust
each broker’s share to match where the LKC’s traffic goes. No broker’s
share can exceed the broker-level maximum.

For more information about moving partition leaders to new brokers,
see [Step 6](#multi-tenancy-scale-step-6).

<a id="multi-tenancy-scale-step-2"></a>

## Step 2: Update the TLS certificate

Reissue the PKC’s TLS certificate before you scale so that it covers the
hostnames of each new broker. The new brokers then start with a
certificate that already covers them.

CFK assigns ordinals to new brokers sequentially, starting
after the highest existing broker ordinal. For example, scaling from
three to five brokers creates brokers `3` and `4`.

Clients reach a broker through a hostname in the following format:

```text
<lkc-id>-kafka.b<broker-ordinal>.<domain>
```

In this format, `<lkc-id>` is the LKC’s ID, which includes the `lkc-`
prefix. The prefix `b` is the default broker prefix. If you set
`brokerPrefix` in the `Kafka` CR, such as in
`spec.listeners.external.externalAccess.loadBalancer.brokerPrefix`,
use that value instead.

Update the Subject Alternative Name (SAN) entries in the certificate for
the TLS approach you chose:

- **Wildcard SAN:** Add a wildcard SAN for each new broker, such as
  `*.b3.mt.example.com`. A wildcard matches only one DNS label, so
  `*.mt.example.com` doesn’t cover these hostnames.
- **Per-LKC SAN:** For each new broker, add a SAN entry for every LKC on
  the PKC, such as `lkc-x7f2q9-kafka.b3.mt.example.com`.

Reissue the certificate, and then update the certificate `Secret` that
the `Kafka` CR’s external listener references. CFK watches secrets in
`.pem` format and, when the content changes, rolls the existing
brokers with under-replicated partition checks.

Wait for the roll to finish before you continue. To check progress, run the
following command until the `READY` column equals `REPLICAS`:

```bash
kubectl get kafka <pkc-name> -n <pkc-namespace>
```

For more information about the TLS approaches, see the Configure TLS
Certificates and DNS for Multi-Tenant PKCs page.

If CFK generates the certificate (`autoGeneratedCerts: true`) and
`managedCerts` is enabled in the CFK Helm values, update the
`platform.confluent.io/managed-cert-add-sans` annotation on the `Kafka`
CR instead. The annotation holds the complete list of custom SANs, so keep
the existing entries and add the wildcard for each new broker, such as
`*.b3.mt.example.com`. CFK doesn’t apply a changed annotation to the
existing certificate on its own. To make CFK reissue the certificate with
the current SANs, set the `platform.confluent.io/force-managed-cert-renewal`
annotation to `true`:

```bash
kubectl annotate kafka <pkc-name> -n <pkc-namespace> \
  platform.confluent.io/force-managed-cert-renewal="true" --overwrite
```

The new brokers start with the reissued certificate. The existing brokers
keep serving only their own hostnames, which the previous certificate
already covers, so you don’t need to restart them.


<a id="multi-tenancy-scale-step-3"></a>

## Step 3: Increase the broker count

Increase the broker count on the PKC’s `Kafka` CR.

1. Scale the `Kafka` CR:
   ```bash
   kubectl scale kafka <pkc-name> --replicas=<new-broker-count> -n <pkc-namespace>
   ```

   Replace the placeholders with your values:
   - `<pkc-name>`: The name of the PKC’s `Kafka` CR.
   - `<new-broker-count>`: The total number of brokers you want after
     scaling, not the number to add.
   - `<pkc-namespace>`: The namespace of the PKC’s `Kafka` CR.

   You don’t need to wait for the new brokers to become ready before you
   [add their DNS records](#multi-tenancy-scale-step-4). Each new
   broker’s load balancer `Service` gets an external address before the
   broker is ready.
2. Confirm that all brokers are ready:
   ```bash
   kubectl get kafka <pkc-name> -n <pkc-namespace>
   ```

   When the `REPLICAS` and `READY` columns show the same number and
   the `STATUS` column shows `RUNNING`, the new brokers are ready.

<a id="multi-tenancy-scale-step-4"></a>

## Step 4: Add DNS records for the new brokers

For each new broker, create one wildcard DNS record, such as
`*.b3.mt.example.com`, that resolves the hostname of every LKC routed to
that broker, for example, `lkc-x7f2q9-kafka.b3.mt.example.com`. Point the
record to the new broker’s external IP address or load balancer. You don’t
need a record for the broker’s base hostname, and you don’t need to change
the bootstrap wildcard record (`*.mt.example.com`), which already resolves
every LKC’s bootstrap hostname.

To find a new broker’s external address, list the PKC’s `Service` objects
and read the `EXTERNAL-IP` column of the new broker’s load balancer
`Service`:

```bash
kubectl get service -n <pkc-namespace>
```

For example, to add the broker with ordinal `3` (hostname label `b3`)
in the `mt.example.com` domain, create the following record:

| DNS name              | Target                 |
|-----------------------|------------------------|
| `*.b3.mt.example.com` | Broker 3 load balancer |

For more information about DNS records for multi-tenant PKCs, see the
Configure TLS Certificates and DNS for Multi-Tenant PKCs page.


Until you create a broker’s wildcard record, the bootstrap wildcard record
(`*.mt.example.com`) also matches the new broker’s hostnames. They then
resolve to the bootstrap load balancer instead of the new broker, so a
successful DNS lookup doesn’t confirm that the new record is in place.

<a id="multi-tenancy-scale-step-5"></a>

## Step 5 (optional): Verify the in-cluster names for each LKC

Applications inside the Kubernetes cluster reach an LKC through
in-cluster names. If `spec.multiTenant.createTenantNameServices` is
`true` on the PKC’s `Kafka` CR, complete this step. Avoid this setting
unless you can’t edit your cluster’s DNS, because the per-broker
`Service` objects use IP addresses from the cluster’s `Service` IP
range. For more information, see [Configure TLS Certificates and DNS for Multi-Tenant PKCs](tls-options.md#multi-tenancy-tls-options).

If the setting is `false`, skip this step. If you configured the CoreDNS
rewrite rules from the TLS options page, no change is needed, because the
rules match any broker number. Otherwise, extend your in-cluster name
resolution to cover the new brokers.

If the setting is `true`, CFK creates a Kubernetes `Service` for each
new broker in every LKC. Verify the `Service` objects by completing the
following steps.

1. List the `Service` objects for an LKC:
   ```bash
   kubectl get service -l platform.confluent.io/logical-cluster-id=<lkc-id> -n <pkc-namespace>
   ```

   Replace the placeholders with your values:
   - `<lkc-id>`: The value of the `status.logicalClusterID` field in
     the `LogicalKafkaCluster` CR.
   - `<pkc-namespace>`: The namespace of the PKC’s `Kafka` CR. CFK
     creates these `Service` objects in that namespace, even if the LKC
     is in a different one.
2. Confirm that the output lists a `Service` named
   `<lkc-id>-kafka-b<broker-ordinal>` for each new broker, in addition
   to the `Service` objects for the existing brokers and the
   `<lkc-id>-kafka` bootstrap `Service`.

   CFK reconciles these `Service` objects periodically, so allow a few
   minutes. If a `Service` is still missing, check the CFK logs.
3. Repeat the previous steps for each LKC on the PKC.

<a id="multi-tenancy-scale-step-6"></a>

## Step 6: Monitor the rebalance onto the new brokers

CFK enables Self-Balancing Clusters by default, including on multi-tenant PKCs. By
default, Self-Balancing Clusters start a rebalance when brokers are added and move
partitions onto the new brokers. A new broker starts with no data and
leads no partitions, so it adds no throughput for any LKC until data
moves to it.

Monitor the rebalance until it completes by using the
`kafka-rebalance-cluster` command. For more information, see
[Monitor the balancer with kafka-rebalance-cluster](../../clusters/sbc/configuration-options.md#sbc-kafka-rebalance-cluster). For more information about how CFK
scales Kafka clusters, see [Scale Confluent Platform Clusters and Balance
Data using Confluent for Kubernetes](https://docs.confluent.io/operator/current/co-scale-cluster.html).

<a id="multi-tenancy-scale-verify"></a>

## Verify the scaled PKC

1. Confirm that the `Kafka` CR shows all brokers ready after scaling
   (`REPLICAS` equals `READY`):
   ```bash
   kubectl get kafka <pkc-name> -n <pkc-namespace>
   ```
2. If you have client credentials for an LKC, use the endpoint in the
   LKC’s `status.bootstrapEndpoint` field to produce to or consume
   from a topic partition whose leader is on a new broker. A successful
   connection confirms that the certificate and DNS records are correct.
3. In Confluent Control Center, open the **Quota Overview** page and confirm that each
   LKC’s usage is within its quota.


## Related content

- Deploy a Multi-Tenant Physical Kafka Cluster
- Configure TLS Certificates and DNS for Multi-Tenant PKCs
- Monitor and Manage Per-LKC Resource Quotas
- Onboard an LKC to a Multi-Tenant Physical Kafka Cluster