<a id="co-multi-az"></a>

# Multiple Availability Zones Deployment of Confluent Platform in Confluent for Kubernetes

Confluent for Kubernetes (CFK) supports deploying Confluent Platform in a regional configuration, where the
Confluent Platform clusters are spread across multiple availability zones (multi-AZ).

This topic describes the behavior and configuration for CFK in multi-AZ
deployments.

To deploy CFK across multiple AZs, first [configure CFK for Rack
Awareness](co-configure-rack-awareness.md#co-configure-rack-awareness), where the racks are AZs. With Rack
Awareness, Kafka partition replicas will be placed across AZs to minimize data
loss in the event of an AZ failure.

## Multi-AZ Kubernetes

In a multi-AZ Kubernetes, the Kubernetes worker nodes are spread across AZs.
Each node is labeled with the region that it is in. To check the AZ labels:

```bash
kubectl get nodes --show-labels
```

An output from the above command includes the AZ label,
`topology.kubernetes.io/zone`. For example:

```bash
topology.kubernetes.io/zone=us-central1-f
```

The storage used by worker nodes is available across all AZs.

## Confluent pods scheduling

The Confluent component instance pods are scheduled across the AZs in a
round-robin fashion. However, CFK does not guarantee that Confluent Platform pods are evenly
scheduled across AZs. For example, when 3 AZs exist, 3 pods might get scheduled
across only 2 zones. Use the topology spread constraints setting in Confluent Platform
component CRs to evenly spread component pods across AZs.

With the following example snippet of a Kafka CR, the number of Kafka pods per AZ
is only allowed to skew by one. An additional Kafka pod can be added to a single
AZ only after all availability zones have an equal number of Kafka pods.

```yaml
kind: Kafka
spec:
  podTemplate:
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: kafka
```

If the AZ does not have enough resources/worker capacity, the pods are scheduled
on any worker in a zone that has capacity.

For example, if you schedule 6 Kafka brokers on a Kubernetes cluster with the
following worker node spread:

* 4 workers in AZ 1
* 1 worker in AZ 2
* 4 workers in AZ 3

The pods would scheduled as one of the following:

* 2 broker pods in AZ 1, 1 broker pod in AZ 2, 3 broker pods in AZ 3
* 3 broker pods in AZ 1, 1 broker pod in AZ 2, 2 broker pods in AZ 3

Once deployed, you can check which nodes your pods are scheduled to:

1. Get the node names of the pods:
   ```bash
   kubectl get pods -o wide
   ```
2. For each node, check which AZ the node is in by looking at the
   `topology.kubernetes.io/zone` label of the node:
   ```bash
   kubectl get nodes --show-labels | grep <node_name>
   ```

## Storage scheduling

CFK uses persistent volumes for each component’s storage needs. In most
cloud providers, the persistent volume storage mechanism is AZ aware, and
persistent volumes cannot be accessed or moved across AZs. Thus, pods that are
scheduled in a specific AZ must have their persistent volume in the same AZ.

The CFK requires lazily bound storage classes on Kubernetes clusters that are
spread across multi-AZs. Ensure that the storage class you use has
`volumeBindingMode` set to `WaitForFirstConsumer`. This delays the binding
and provisioning of the PersistentVolume until a pod using the
PersistentVolumeClaim is created. PersistentVolumes are selected or provisioned
conforming to the topology that is specified by the Pod’s scheduling
constraints.

Check the binding mode of a storage class with the following command:

```bash
kubectl describe sc <your_storage_class>
```

The output should have `VolumeBindingMode` set to `WaitForFirstConsumer` as
shown below:

```bash
Name:                  <your_storage_class>
...
VolumeBindingMode:     WaitForFirstConsumer
...
```

Once deployed, you can check to see which AZ the persistent volume is assigned
to. For example:

```bash
kubectl describe pv pvc-008a81fd-cd28-4b3e-a26f-5fe9384476f9
```

A sample output of the above command that shows the persistent volume is mounted
in the `us-central1-a` zone:

```bash
Name:        pvc-008a81fd-cd28-4b3e-a26f-5fe9384476f9
Labels:      topology.kubernetes.io/region=us-central1
             topology.kubernetes.io/zone=us-central1-a
```

## Scenario: Worker failure

When a single Kubernetes worker node fails, the pods on that worker node
disappear.

In that case, CFK will schedule the Confluent pod on another Kubernetes worker
with the required resource capacity in the same AZ, and will attach to the same
persistent volume.

When there is no available worker node with the required resource
capacity, the pod will not be scheduled, and it will be in the pending state.

When resource capacity becomes available in the zone, the pod will be scheduled.

#### NOTE
CFK uses the oneReplicaPerNode feature to ensure that Kafka brokers are
not co-located on the same Kubernetes worker node, and the same for ZooKeeper
servers. Therefore, you need to have enough Kubernetes worker nodes to
accommodate Kafka and ZooKeeper.

## Scenario: AZ failure

When an entire AZ fails, all the Kubernetes worker nodes in that AZ are unavailable. The
storage is resilient within the AZ, and thus is not available as well.

In this case, the pods and the persistent volumes are not available, and they
cannot be brought up automatically. They will be in the pending state.

There are two paths that you can take to resolve the issue: waiting for AZ to
become available or scheduling pods to a different AZ.

### Wait for AZ to become available

When you wait for the AZ to be available, the specific pods in that AZ will
remain in the pending state until the AZ is available. When the AZ is back up,
those pods will be brought up.

In this case, make sure that there is enough broker capacity to operate in case
of an AZ being down. The replication factor and ISR (In-Sync Replicas) should
be set to work with the number of brokers available when one AZ is out.

<a id="co-multi-az-recover-broker-volume-loss"></a>

### Recover a broker after volume loss

Use the following procedure to recover a broker that lost its persistent
volume, for example due to an AZ or node failure. The other brokers keep
running and aren’t restarted.

The example commands below use `<namespace>` as a placeholder for your
namespace, `kafka` as the Kafka cluster name, and `0` as the ordinal of
the lost broker.

To recover the broker:

1. Record the lost broker’s volume name, for reference:
   ```bash
   kubectl -n <namespace> get pvc data0-kafka-0 -o jsonpath='{.spec.volumeName}{"\n"}'
   ```
2. Temporarily block the placeholder storage class from provisioning a
   `pvc`:
   ```bash
   kubectl -n <namespace> create resourcequota block-cfk-placeholder-pvc \
     --hard=dummy.storageclass.storage.k8s.io/persistentvolumeclaims=0
   ```
3. Pause the operator for this cluster:
   ```bash
   kubectl -n <namespace> annotate kafka kafka platform.confluent.io/block-reconcile=true --overwrite
   ```
4. Delete the broker pod, then confirm the replacement pod is `Pending`:
   ```bash
   kubectl -n <namespace> delete pod kafka-0 --grace-period=0 --force
   kubectl -n <namespace> get pod kafka-0
   ```
5. Delete the `pvc`, then confirm it returns `NotFound`:
   ```bash
   kubectl -n <namespace> delete pvc data0-kafka-0 --wait=false
   kubectl -n <namespace> get pvc data0-kafka-0
   ```

   `StatefulSet` `FailedCreate` events mentioning `exceeded quota:
   block-cfk-placeholder-pvc` are expected at this point.
6. Resume the operator:
   ```bash
   kubectl -n <namespace> annotate kafka kafka platform.confluent.io/block-reconcile-
   ```
7. Verify the broker recovered:
   ```bash
   kubectl -n <namespace> get pvc data0-kafka-0
   kubectl -n <namespace> get pod kafka-0 -o wide
   kubectl -n <namespace> get kafka kafka
   ```

   Confirm that `data0-kafka-0` is `Bound` with the storage class and
   size from the Kafka CR, `kafka-0` is `Running` and `Ready`, no
   under-replicated partitions remain, and the Kafka CR status is
   `RUNNING`.
8. Delete the quota:
   ```bash
   kubectl -n <namespace> delete resourcequota block-cfk-placeholder-pvc
   ```