<a id="co-disaster-recovery"></a>

# Disaster Recovery for Multi-Region KRaft Clusters

This topic explains how to recover a multi-region Confluent Platform cluster that uses a
dynamic KRaft quorum after a region failure causes the cluster to lose the
controller quorum.

When a region in a multi-region cluster (MRC) goes down, it can take enough
KRaft controllers with it that the metadata quorum can no longer elect a
leader. When this happens, the cluster blocks metadata writes and cannot
make progress, even though the controllers and brokers in the surviving region
are still running.

This procedure rebuilds a working quorum from the controllers in the surviving
region so that the cluster becomes available again. After recovery, the cluster
runs with reduced fault tolerance until you restore the failed region.

CFK provides two recovery paths:

* **Confluent kubectl plugin (recommended)**: The `kubectl confluent kraft`
  plugin rebuilds the surviving region’s quorum in two commands and records its
  progress so that an interrupted recovery can resume.
* **Manual recovery**: You run the same sequence of steps by hand. Use this path
  when you cannot use the plugin, or when you want full control over each step.

Both paths require CFK 3.3 or later. If you use CFK 3.2, see the
[disaster recovery example for CFK 3.2](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/kraft/dynamic-quorum/disaster-recovery/quorum-loss-recovery/cfk-3.2)
instead.

## Quorum loss in a multi-region cluster

In a dynamic KRaft quorum, the controllers that vote on metadata changes are
called *voters*. The quorum can elect a leader and accept metadata writes only
while a majority of voters are available.

Consider a two-region cluster with six controller voters, three in each region.
The majority is four voters. If one region fails, only three voters remain,
which is less than the majority, so the quorum has no leader and metadata
writes are blocked.

In this example, the three failed voters are fewer than the majority, so every
committed metadata write reached at least one surviving controller. Rebuilding
the quorum from the surviving controllers loses no metadata, giving a Recovery
Point Objective (RPO) of zero.

This procedure also applies when the number of failed voters is equal to or
greater than the majority. In that case, a committed metadata write might exist
only on failed controllers, so rebuilding the quorum can lose that metadata.

#### IMPORTANT
If the failed voters are a majority, weigh the tradeoff between
availability and data loss before you proceed. If the failed region might
come back, wait for it instead of running recovery. If the region is
permanently gone, or you cannot estimate when it will return, you can still
run this procedure, understanding that metadata committed only to the
failed controllers is lost. Contact
[Confluent Support](https://support.confluent.io/) to plan the recovery.

## Before you begin

### Prerequisites

* CFK 3.3 or later, with the operator and init image on the same version.
* A dynamic KRaft quorum enabled on the `KRaftController` custom resource
  (CR). For details, see [Configure Dynamic KRaft Quorum for Confluent Platform Using Confluent for Kubernetes](co-configure-kraft-dynamic-quorum.md#co-configure-kraft-dynamic-quorum).
* Exactly one `KRaftController` custom resource (CR) in the target
  namespace, regardless of how many controller pods that CR defines.
* For the plugin path, the Confluent kubectl plugin (`kubectl-confluent`)
  installed. Verify that it is available:
  ```bash
  kubectl confluent kraft -h
  ```
* All KRaft and Kafka persistent volumes (PVs) configured with
  `reclaimPolicy: Retain` as a precaution.

### Keep the failed region down during recovery

Recovery rebuilds the quorum from scratch, so the failed region must stay
offline for the entire process. Recovery creates a new metadata timeline. If
the failed region’s controllers come back while recovery is in progress and
can reach each other, they can form a second, competing quorum, which results
in split-brain, where two quorums disagree about cluster metadata. This
behavior is inherent to how KRaft works and is not specific to CFK.

#### WARNING
Keep the failed region unreachable, cordoned, or scaled to zero until you
finish recovering the surviving region.

### Snapshot the KRaft volumes

Before you start recovery, take a volume-level snapshot of each surviving
controller’s KRaft PV. This snapshot is your only reliable rollback if
recovery goes wrong, for example if you rebuild from the wrong controller.

The following example takes a snapshot of a controller’s PV on Google
Kubernetes Engine (GKE). Repeat it for each surviving controller.

```bash
PV=$(kubectl --context <surviving-context> -n <namespace> get pvc data0-kraftcontroller-0 -o jsonpath='{.spec.volumeName}')
DISK=$(kubectl --context <surviving-context> get pv "$PV" -o jsonpath='{.spec.csi.volumeHandle}' | awk -F/ '{print $NF}')
gcloud compute snapshots create dr-presnap-kraftcontroller-0-$(date +%s) --source-disk "$DISK" --source-disk-zone <zone>
```

## Recovery overview

Recovery has two phases:

* **Phase 1: Recover the surviving region.** Rebuild a working quorum from the
  controllers that are still running. When Phase 1 completes, the cluster is
  available again on a smaller quorum. Use either the plugin or the manual
  path.
* **Phase 2: Restore the failed region.** Bring the failed region back into
  the quorum. Phase 2 is a manual procedure in both paths and is not urgent
  after Phase 1 is complete.

In both paths, Phase 1 rebuilds the quorum from a single *seed* controller. The
seed is the surviving controller with the most up-to-date metadata log.

#### IMPORTANT
Choose the seed by the highest metadata epoch first, and break ties by the
largest log end offset. Do not choose the seed by log end offset alone. A
longer log with a lower epoch can be a stale branch that is missing committed
metadata. Seeding from it permanently loses that metadata.

<a id="co-disaster-recovery-phase-1"></a>

## Recover the surviving region (Phase 1)

Choose the path that matches your environment. The plugin path is recommended
because it runs the entire sequence in two commands and can resume after an
interruption.

### Confluent kubectl plugin (recommended)

The plugin rebuilds the surviving region’s quorum in two commands. It
parks the controllers, reads each controller’s metadata position,
rebuilds the quorum from the seed, and rejoins the remaining controllers.

1. Read the metadata position of each surviving controller. This command
   annotates the controllers into maintenance mode, restarts them so the
   annotation takes effect, and then reads each controller’s metadata
   log length while its KRaft process is not running.
   ```bash
   kubectl confluent kraft log-length --context <surviving-context> -n <namespace>
   ```

   The command prints the metadata epoch and log end offset for each
   controller.
2. From the output, choose the seed controller. Pick the controller with
   the highest epoch. If more than one controller has the highest epoch,
   pick the one with the largest log end offset.

   #### NOTE
   The command’s output currently suggests the controller with the
   largest log end offset without accounting for epoch. Use the log
   end offset to break ties only when epochs match. Always prefer the
   highest epoch first, even if another controller has a larger log
   end offset.
3. Rebuild the quorum from the seed:
   ```bash
   kubectl confluent kraft recover-region --context <surviving-context> -n <namespace> --seed-pod <seed-pod>
   ```

   The command snapshots each controller’s metadata, rebuilds the quorum
   from the seed, and rejoins the remaining controllers as voters. When it
   finishes, it prints the final quorum status.

The following flags are available on the `recover-region` and
`log-length` commands:

| Flag                      | Command          | Description                                                                                                                                                             |
|---------------------------|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--seed-pod`              | `recover-region` | Required. The seed controller you chose from the `log-length`<br/>output.                                                                                               |
| `--yes`                   | both             | Skip the interactive confirmation prompt.                                                                                                                               |
| `--timeout`               | both             | Overall time budget. The default is 5 minutes for `log-length`<br/>and 15 minutes for `recover-region`.                                                                 |
| `--skip-backup`           | `recover-region` | Skip the pre-recovery in-pod metadata snapshot. Use this flag when<br/>you already took a PV snapshot or when the volume lacks space.                                   |
| `--backup-dir`            | `recover-region` | Directory to store the pre-recovery in-pod metadata snapshot.<br/>Defaults to a `backup` directory next to the metadata log.<br/>Ignored when you pass `--skip-backup`. |
| `--force-standalone-done` | `recover-region` | Resume after a rebuild step that you completed manually with<br/>Confluent Support. See [Handle a failed recovery](#co-disaster-recovery-failed).                       |
| `--metadata-log-dir`      | both             | Override the auto-detected metadata log directory.                                                                                                                      |

**How the plugin records progress**

The plugin records its progress to a ConfigMap named
`<kraftcontroller-name>-dr-checkpoint` in the controller’s namespace, so
an interrupted recovery can resume instead of starting over. To inspect
the progress, view the ConfigMap:

```bash
kubectl get configmap <kraftcontroller-name>-dr-checkpoint -n <namespace> \
  -o jsonpath='{.data.checkpoint\.json}' | jq
```

Do not edit the ConfigMap by hand. To control a recovery, re-run the
command, which resumes automatically. Delete the ConfigMap only after
you confirm the region is fully recovered, as part of cleanup.

### Manual recovery

In the manual path, you run the same sequence by hand. The examples use a
two-region cluster with three controllers in the surviving region, named
`kraftcontroller-0`, `kraftcontroller-1`, and `kraftcontroller-2` in
the `<namespace>` namespace. Adapt the pod names, namespace, and
controller count to your topology.

Because the rebuild step is irreversible, the recommended approach is to
run the manual path with Confluent Support.

The commands use the Kafka client configuration that CFK 3.3 mounts on the
controller pods, and the metadata log directory on the pod:

```bash
CFG=/opt/confluentinc/etc/kafka/kafka-client.properties
LOGDIR=/mnt/data/data0/logs
```

1. Park every surviving controller in maintenance mode. In maintenance
   mode, the operator restarts each pod and the main container sleeps
   before starting the KRaft process, so the controller does not serve
   or modify metadata and its metadata log is free for the recovery tool.
   ```bash
   kubectl annotate kraftcontroller kraftcontroller -n <namespace> \
     platform.confluent.io/maintenance-mode="kraftcontroller-0,kraftcontroller-1,kraftcontroller-2" --overwrite
   kubectl delete pod kraftcontroller-0 kraftcontroller-1 kraftcontroller-2 -n <namespace> --grace-period=0 --force
   ```

   Wait until each pod’s main container log shows the
   `MAINTENANCE mode (main container)` message.
2. Measure the metadata epoch and log end offset on each parked
   controller:
   ```bash
   for p in kraftcontroller-0 kraftcontroller-1 kraftcontroller-2; do
     kubectl exec $p -n <namespace> -c kraftcontroller -- bash -c \
       '[ -f '"$LOGDIR"'/.lock ] || : > '"$LOGDIR"'/.lock; \
        kafka-metadata-recovery reconfig log-length --metadata-log-dir '"$LOGDIR"
   done
   ```
3. Choose the seed controller. Pick the controller with the highest
   epoch, and on a tie, the one with the largest log end offset.
4. Optionally, back up the metadata partition on each controller as an
   extra safeguard. If you already took a PV snapshot, you can skip
   this step.
   ```bash
   for p in kraftcontroller-0 kraftcontroller-1 kraftcontroller-2; do
     kubectl exec $p -n <namespace> -c kraftcontroller -- bash -c \
       'mkdir -p '"$LOGDIR"'/backup && cp -r '"$LOGDIR"'/__cluster_metadata-0 '"$LOGDIR"'/backup/__cluster_metadata-0-$(date +%s) 2>/dev/null || true'
   done
   ```
5. Rebuild the voter set on the seed. This step is irreversible. Run it
   once, only on the seed. The following example assumes the seed is
   `kraftcontroller-1`.
   ```bash
   kubectl exec kraftcontroller-1 -n <namespace> -c kraftcontroller -- bash -c \
     '[ -f '"$LOGDIR"'/.lock ] || : > '"$LOGDIR"'/.lock; \
      kafka-metadata-recovery reconfig force-standalone --config '"$CFG"
   ```
6. Release the seed from maintenance mode and restart it. It comes back as
   the leader of a single-voter quorum.
   ```bash
   kubectl annotate kraftcontroller kraftcontroller -n <namespace> \
     platform.confluent.io/maintenance-mode="kraftcontroller-0,kraftcontroller-2" --overwrite
   kubectl delete pod kraftcontroller-1 -n <namespace> --grace-period=0 --force
   ```
7. For each remaining controller, clear its stale metadata, release it
   from maintenance mode so it starts empty and joins as an observer, and
   then add it back as a voter.

   First, verify that the seed is running and has become the quorum
   leader:
   ```bash
   kubectl exec kraftcontroller-1 -n <namespace> -c kraftcontroller -- kafka-metadata-quorum \
     --command-config $CFG --bootstrap-controller <controller-bootstrap-servers> describe --status
   ```

   Then clear the stale metadata and release the controllers from
   maintenance mode:
   ```bash
   for ord in 0 2; do
     kubectl exec kraftcontroller-$ord -n <namespace> -c kraftcontroller -- bash -c \
       'rm -rf '"$LOGDIR"'/__cluster_metadata-0'
   done
   kubectl annotate kraftcontroller kraftcontroller -n <namespace> platform.confluent.io/maintenance-mode-
   kubectl delete pod kraftcontroller-0 kraftcontroller-2 -n <namespace> --grace-period=0 --force
   ```

   Wait until each restarted pod is running and the KRaft process is
   ready to accept commands before proceeding.
   ```bash
   kubectl wait pod kraftcontroller-0 kraftcontroller-2 -n <namespace> --for=condition=Ready --timeout=300s
   for ord in 0 2; do
     kubectl exec kraftcontroller-$ord -n <namespace> -c kraftcontroller -- kafka-metadata-quorum \
       --command-config $CFG --bootstrap-controller <controller-bootstrap-servers> add-controller
   done
   ```

Phase 1 is complete. The cluster is operational on the surviving region’s
quorum.

## Restore the failed region (Phase 2)

After Phase 1, the cluster runs on the surviving region’s quorum with reduced
fault tolerance. Phase 2 brings the failed region back and restores full
resilience. This step is not urgent and follows the same manual procedure
whether you used the plugin or the manual path for Phase 1.

To restore the failed region, do the following for its controllers:

1. Set the maintenance-mode annotation for the failed region’s controllers
   before you bring them back, so they come up parked.
2. Bring the region back, for example by uncordoning its nodes or scaling its
   controllers back up.
3. Move the stale metadata partition aside on each returning controller.
4. Release each controller from maintenance mode. It starts empty, fetches
   metadata from the recovered leader, and joins as an observer.
5. Add each controller back as a voter.

If the failed region held fewer than a majority of voters, the surviving region
holds every committed metadata write, so the metadata moved aside on the
returning controllers can be discarded. If the failed region held a majority
of voters, recovery might have lost some committed metadata. Contact
[Confluent Support](https://support.confluent.io/) if you need to confirm
what to keep before discarding the moved-aside metadata. Kafka topic data on
the returning region is preserved regardless. Brokers reattach to their
retained log volumes and rejoin the in-sync replica set through the normal
Kafka mechanism.

## Clean up after recovery

Do not clean up until you confirm that the full quorum is healthy. Verify the
quorum with the following command:

```bash
kubectl exec kraftcontroller-0 -n <namespace> -c kraftcontroller -- kafka-metadata-quorum \
  --command-config $CFG --bootstrap-controller <controller-bootstrap-servers> describe --status
```

After you confirm the quorum is healthy, clean up the recovery artifacts:

* Delete the in-disk metadata backups on each controller:
  ```bash
  for p in kraftcontroller-0 kraftcontroller-1 kraftcontroller-2; do
    kubectl exec $p -n <namespace> -c kraftcontroller -- bash -c 'rm -rf /mnt/data/data0/logs/backup'
  done
  ```
* Delete the PV snapshots you took before recovery.
* If you used the plugin, delete the checkpoint ConfigMap:
  ```bash
  kubectl delete configmap <kraftcontroller-name>-dr-checkpoint -n <namespace>
  ```

<a id="co-disaster-recovery-failed"></a>

## Handle a failed recovery

If recovery is interrupted or a step fails, the controllers remain parked
so you can inspect the state. Start by checking which step failed.

If you used the plugin, inspect the checkpoint ConfigMap to see which step
completed last:

```bash
kubectl get configmap <kraftcontroller-name>-dr-checkpoint -n <namespace> \
  -o jsonpath='{.data.checkpoint\.json}' | jq
```

If you used the manual path, the last command you ran indicates which step
failed.

### Retry a failed step

* If `log-length` failed, rerun it. The command is idempotent, and a failure
  is usually transient.
* If `recover-region` failed on any step other than the voter-set rebuild
  (`force-standalone`), for example parking, snapshotting, clearing stale
  metadata, releasing from maintenance mode, or adding a controller, rerun the
  same command with the same `--seed-pod`. These steps are safe to redo. Do
  not change `--seed-pod` between runs.
* If `recover-region` failed during the voter-set rebuild step, see the
  following warning.

#### WARNING
If the voter-set rebuild step (`force-standalone`) fails or is interrupted,
do not rerun it. It might have already partially rewritten the voter set, and
rerunning it can corrupt the metadata state. Contact
[Confluent Support](https://support.confluent.io/) to confirm whether the
step completed.

For the plugin path, resume recovery after you confirm the status of the
voter-set rebuild step:

* If Confluent Support confirms the step completed, or the command itself
  succeeded but the plugin failed afterward, for example while writing the
  checkpoint, resume with the `--force-standalone-done` flag so the plugin
  records the step as done and continues from the next step:
  ```bash
  kubectl confluent kraft recover-region --context <surviving-context> -n <namespace> --seed-pod <seed-pod> --force-standalone-done
  ```

If you already took a volume-level PV snapshot before recovery, pass
`--skip-backup` on `recover-region` to avoid the unnecessary in-pod
metadata copy.

If an environment issue prevents you from using the plugin at all, continue
manually. Check the checkpoint ConfigMap to see which steps already completed,
then continue with the corresponding steps in the manual path.