Disaster Recovery for Multi-Region KRaft Clusters

This topic explains how to recover a multi-region Confluent Platform cluster that uses a dynamic KRaft quorum after a region failure causes the cluster to lose the controller quorum.

When a region in a multi-region cluster (MRC) goes down, it can take enough KRaft controllers with it that the metadata quorum can no longer elect a leader. When this happens, the cluster blocks metadata writes and cannot make progress, even though the controllers and brokers in the surviving region are still running.

This procedure rebuilds a working quorum from the controllers in the surviving region so that the cluster becomes available again. After recovery, the cluster runs with reduced fault tolerance until you restore the failed region.

CFK provides two recovery paths:

• Confluent kubectl plugin (recommended): The kubectl confluent kraft plugin rebuilds the surviving region’s quorum in two commands and records its progress so that an interrupted recovery can resume.

• Manual recovery: You run the same sequence of steps by hand. Use this path when you cannot use the plugin, or when you want full control over each step.

Both paths require CFK 3.3 or later. If you use CFK 3.2, see the disaster recovery example for CFK 3.2 instead.

Quorum loss in a multi-region cluster

In a dynamic KRaft quorum, the controllers that vote on metadata changes are called voters. The quorum can elect a leader and accept metadata writes only while a majority of voters are available.

Consider a two-region cluster with six controller voters, three in each region. The majority is four voters. If one region fails, only three voters remain, which is less than the majority, so the quorum has no leader and metadata writes are blocked.

In this example, the three failed voters are fewer than the majority, so every committed metadata write reached at least one surviving controller. Rebuilding the quorum from the surviving controllers loses no metadata, giving a Recovery Point Objective (RPO) of zero.

This procedure also applies when the number of failed voters is equal to or greater than the majority. In that case, a committed metadata write might exist only on failed controllers, so rebuilding the quorum can lose that metadata.

Important

If the failed voters are a majority, weigh the tradeoff between availability and data loss before you proceed. If the failed region might come back, wait for it instead of running recovery. If the region is permanently gone, or you cannot estimate when it will return, you can still run this procedure, understanding that metadata committed only to the failed controllers is lost. Contact Confluent Support to plan the recovery.

Before you begin

Prerequisites

• CFK 3.3 or later, with the operator and init image on the same version.

• A dynamic KRaft quorum enabled on the KRaftController custom resource (CR). For details, see Configure Dynamic KRaft Quorum for Confluent Platform Using Confluent for Kubernetes.

• Exactly one KRaftController custom resource (CR) in the target namespace, regardless of how many controller pods that CR defines.

• For the plugin path, the Confluent kubectl plugin (kubectl-confluent) installed. Verify that it is available:

  kubectl confluent kraft -h
  

• All KRaft and Kafka persistent volumes (PVs) configured with reclaimPolicy: Retain as a precaution.

Keep the failed region down during recovery

Recovery rebuilds the quorum from scratch, so the failed region must stay offline for the entire process. Recovery creates a new metadata timeline. If the failed region’s controllers come back while recovery is in progress and can reach each other, they can form a second, competing quorum, which results in split-brain, where two quorums disagree about cluster metadata. This behavior is inherent to how KRaft works and is not specific to CFK.

Warning

Keep the failed region unreachable, cordoned, or scaled to zero until you finish recovering the surviving region.

Snapshot the KRaft volumes

Before you start recovery, take a volume-level snapshot of each surviving controller’s KRaft PV. This snapshot is your only reliable rollback if recovery goes wrong, for example if you rebuild from the wrong controller.

The following example takes a snapshot of a controller’s PV on Google Kubernetes Engine (GKE). Repeat it for each surviving controller.

PV=$(kubectl --context <surviving-context> -n <namespace> get pvc data0-kraftcontroller-0 -o jsonpath='{.spec.volumeName}')
DISK=$(kubectl --context <surviving-context> get pv "$PV" -o jsonpath='{.spec.csi.volumeHandle}' | awk -F/ '{print $NF}')
gcloud compute snapshots create dr-presnap-kraftcontroller-0-$(date +%s) --source-disk "$DISK" --source-disk-zone <zone>

Recovery overview

Recovery has two phases:

• Phase 1: Recover the surviving region. Rebuild a working quorum from the controllers that are still running. When Phase 1 completes, the cluster is available again on a smaller quorum. Use either the plugin or the manual path.

• Phase 2: Restore the failed region. Bring the failed region back into the quorum. Phase 2 is a manual procedure in both paths and is not urgent after Phase 1 is complete.

In both paths, Phase 1 rebuilds the quorum from a single seed controller. The seed is the surviving controller with the most up-to-date metadata log.

Important

Choose the seed by the highest metadata epoch first, and break ties by the largest log end offset. Do not choose the seed by log end offset alone. A longer log with a lower epoch can be a stale branch that is missing committed metadata. Seeding from it permanently loses that metadata.

Recover the surviving region (Phase 1)

Choose the path that matches your environment. The plugin path is recommended because it runs the entire sequence in two commands and can resume after an interruption.

Tip

Test this recovery procedure, using either path, in a non-production environment before you need it during an actual disaster. Practicing the procedure surfaces environment-specific quirks so you know how to handle them when it matters.

Confluent kubectl plugin (recommended)

The plugin rebuilds the surviving region’s quorum in two commands. It parks the controllers, reads each controller’s metadata position, rebuilds the quorum from the seed, and rejoins the remaining controllers.

1. Read the metadata position of each surviving controller. This command annotates the controllers into maintenance mode, restarts them so the annotation takes effect, and then reads each controller’s metadata log length while its KRaft process is not running.

   kubectl confluent kraft log-length --context <surviving-context> -n <namespace>
   

   The command prints the metadata epoch and log end offset for each controller.

2. From the output, choose the seed controller. Pick the controller with the highest epoch. If more than one controller has the highest epoch, pick the one with the largest log end offset.

   Note

   The command’s output currently suggests the controller with the largest log end offset without accounting for epoch. Use the log end offset to break ties only when epochs match. Always prefer the highest epoch first, even if another controller has a larger log end offset.

3. Rebuild the quorum from the seed:

   kubectl confluent kraft recover-region --context <surviving-context> -n <namespace> --seed-pod <seed-pod>
   

   The command snapshots each controller’s metadata, rebuilds the quorum from the seed, and rejoins the remaining controllers as voters. When it finishes, it prints the final quorum status.

The following flags are available on the recover-region and log-length commands:

Flag	Command	Description
--seed-pod	recover-region	Required. The seed controller you chose from the log-length output.
--yes	both	Skip the interactive confirmation prompt.
--timeout	both	Overall time budget. The default is 5 minutes for log-length and 15 minutes for recover-region.
--skip-backup	recover-region	Skip the pre-recovery in-pod metadata snapshot. Use this flag when you already took a PV snapshot or when the volume lacks space.
--backup-dir	recover-region	Directory to store the pre-recovery in-pod metadata snapshot. Defaults to a backup directory next to the metadata log. Ignored when you pass --skip-backup.
--force-standalone-done	recover-region	Resume after a rebuild step that you completed manually with Confluent Support. See Handle a failed recovery.
--metadata-log-dir	both	Override the auto-detected metadata log directory.

How the plugin records progress

The plugin records its progress to a ConfigMap named <kraftcontroller-name>-dr-checkpoint in the controller’s namespace, so an interrupted recovery can resume instead of starting over. To inspect the progress, view the ConfigMap:

kubectl get configmap <kraftcontroller-name>-dr-checkpoint -n <namespace> \
  -o jsonpath='{.data.checkpoint\.json}' | jq

Do not edit the ConfigMap by hand. To control a recovery, re-run the command, which resumes automatically. Delete the ConfigMap only after you confirm the region is fully recovered, as part of cleanup.

Manual recovery

In the manual path, you run the same sequence by hand. The examples use a two-region cluster with three controllers in the surviving region, named kraftcontroller-0, kraftcontroller-1, and kraftcontroller-2 in the <namespace> namespace. Adapt the pod names, namespace, and controller count to your topology.

Because the rebuild step is irreversible, the recommended approach is to run the manual path with Confluent Support.

The commands use the Kafka client configuration that CFK 3.3 mounts on the controller pods, and the metadata log directory on the pod:

CFG=/opt/confluentinc/etc/kafka/kafka-client.properties
LOGDIR=/mnt/data/data0/logs

1. Park every surviving controller in maintenance mode. In maintenance mode, the operator restarts each pod and the main container sleeps before starting the KRaft process, so the controller does not serve or modify metadata and its metadata log is free for the recovery tool.

   kubectl annotate kraftcontroller kraftcontroller -n <namespace> \
     platform.confluent.io/maintenance-mode="kraftcontroller-0,kraftcontroller-1,kraftcontroller-2" --overwrite
   kubectl delete pod kraftcontroller-0 kraftcontroller-1 kraftcontroller-2 -n <namespace> --grace-period=0 --force
   

   Wait until each pod’s main container log shows the MAINTENANCE mode (main container) message.

2. Measure the metadata epoch and log end offset on each parked controller:

   for p in kraftcontroller-0 kraftcontroller-1 kraftcontroller-2; do
     kubectl exec $p -n <namespace> -c kraftcontroller -- bash -c \
       '[ -f '"$LOGDIR"'/.lock ] || : > '"$LOGDIR"'/.lock; \
        kafka-metadata-recovery reconfig log-length --metadata-log-dir '"$LOGDIR"
   done
   

3. Choose the seed controller. Pick the controller with the highest epoch, and on a tie, the one with the largest log end offset.

4. Optionally, back up the metadata partition on each controller as an extra safeguard. If you already took a PV snapshot, you can skip this step.

   for p in kraftcontroller-0 kraftcontroller-1 kraftcontroller-2; do
     kubectl exec $p -n <namespace> -c kraftcontroller -- bash -c \
       'mkdir -p '"$LOGDIR"'/backup && cp -r '"$LOGDIR"'/__cluster_metadata-0 '"$LOGDIR"'/backup/__cluster_metadata-0-$(date +%s) 2>/dev/null || true'
   done
   

5. Rebuild the voter set on the seed. This step is irreversible. Run it once, only on the seed. The following example assumes the seed is kraftcontroller-1.

   kubectl exec kraftcontroller-1 -n <namespace> -c kraftcontroller -- bash -c \
     '[ -f '"$LOGDIR"'/.lock ] || : > '"$LOGDIR"'/.lock; \
      kafka-metadata-recovery reconfig force-standalone --config '"$CFG"
   

6. Release the seed from maintenance mode and restart it. It comes back as the leader of a single-voter quorum.

   kubectl annotate kraftcontroller kraftcontroller -n <namespace> \
     platform.confluent.io/maintenance-mode="kraftcontroller-0,kraftcontroller-2" --overwrite
   kubectl delete pod kraftcontroller-1 -n <namespace> --grace-period=0 --force
   

7. For each remaining controller, clear its stale metadata, release it from maintenance mode so it starts empty and joins as an observer, and then add it back as a voter.

   First, verify that the seed is running and has become the quorum leader:

   kubectl exec kraftcontroller-1 -n <namespace> -c kraftcontroller -- kafka-metadata-quorum \
     --command-config $CFG --bootstrap-controller <controller-bootstrap-servers> describe --status
   

   Then clear the stale metadata and release the controllers from maintenance mode:

   for ord in 0 2; do
     kubectl exec kraftcontroller-$ord -n <namespace> -c kraftcontroller -- bash -c \
       'rm -rf '"$LOGDIR"'/__cluster_metadata-0'
   done
   kubectl annotate kraftcontroller kraftcontroller -n <namespace> platform.confluent.io/maintenance-mode-
   kubectl delete pod kraftcontroller-0 kraftcontroller-2 -n <namespace> --grace-period=0 --force
   

   Wait until each restarted pod is running and the KRaft process is ready to accept commands before proceeding.

   kubectl wait pod kraftcontroller-0 kraftcontroller-2 -n <namespace> --for=condition=Ready --timeout=300s
   for ord in 0 2; do
     kubectl exec kraftcontroller-$ord -n <namespace> -c kraftcontroller -- kafka-metadata-quorum \
       --command-config $CFG --bootstrap-controller <controller-bootstrap-servers> add-controller
   done
   

Phase 1 is complete. The cluster is operational on the surviving region’s quorum.

Restore the failed region (Phase 2)

After Phase 1, the cluster runs on the surviving region’s quorum with reduced fault tolerance. Phase 2 brings the failed region back and restores full resilience. This step is not urgent and follows the same manual procedure whether you used the plugin or the manual path for Phase 1.

To restore the failed region, do the following for its controllers:

1. Set the maintenance-mode annotation for the failed region’s controllers before you bring them back, so they come up parked.

2. Bring the region back, for example by uncordoning its nodes or scaling its controllers back up.

3. Move the stale metadata partition aside on each returning controller.

4. Release each controller from maintenance mode. It starts empty, fetches metadata from the recovered leader, and joins as an observer.

5. Add each controller back as a voter.

If the failed region held fewer than a majority of voters, the surviving region holds every committed metadata write, so the metadata moved aside on the returning controllers can be discarded. If the failed region held a majority of voters, recovery might have lost some committed metadata. Contact Confluent Support if you need to confirm what to keep before discarding the moved-aside metadata. Kafka topic data on the returning region is preserved regardless. Brokers reattach to their retained log volumes and rejoin the in-sync replica set through the normal Kafka mechanism.

Clean up after recovery

Do not clean up until you confirm that the full quorum is healthy. Verify the quorum with the following command:

kubectl exec kraftcontroller-0 -n <namespace> -c kraftcontroller -- kafka-metadata-quorum \
  --command-config $CFG --bootstrap-controller <controller-bootstrap-servers> describe --status

After you confirm the quorum is healthy, clean up the recovery artifacts:

• Delete the in-disk metadata backups on each controller:

  for p in kraftcontroller-0 kraftcontroller-1 kraftcontroller-2; do
    kubectl exec $p -n <namespace> -c kraftcontroller -- bash -c 'rm -rf /mnt/data/data0/logs/backup'
  done
  

• Delete the PV snapshots you took before recovery.

• If you used the plugin, delete the checkpoint ConfigMap:

  kubectl delete configmap <kraftcontroller-name>-dr-checkpoint -n <namespace>
  

Handle a failed recovery

If recovery is interrupted or a step fails, the controllers remain parked so you can inspect the state. Start by checking which step failed.

If you used the plugin, inspect the checkpoint ConfigMap to see which step completed last:

kubectl get configmap <kraftcontroller-name>-dr-checkpoint -n <namespace> \
  -o jsonpath='{.data.checkpoint\.json}' | jq

If you used the manual path, the last command you ran indicates which step failed.

Retry a failed step

• If log-length failed, rerun it. The command is idempotent, and a failure is usually transient.

• If recover-region failed on any step other than the voter-set rebuild (force-standalone), for example parking, snapshotting, clearing stale metadata, releasing from maintenance mode, or adding a controller, rerun the same command with the same --seed-pod. These steps are safe to redo. Do not change --seed-pod between runs.

• If recover-region failed during the voter-set rebuild step, see the following warning.

Warning

If the voter-set rebuild step (force-standalone) fails or is interrupted, do not rerun it. It might have already partially rewritten the voter set, and rerunning it can corrupt the metadata state. Contact Confluent Support to confirm whether the step completed.

For the plugin path, resume recovery after you confirm the status of the voter-set rebuild step:

• If Confluent Support confirms the step completed, or the command itself succeeded but the plugin failed afterward, for example while writing the checkpoint, resume with the --force-standalone-done flag so the plugin records the step as done and continues from the next step:

  kubectl confluent kraft recover-region --context <surviving-context> -n <namespace> --seed-pod <seed-pod> --force-standalone-done
  

If you already took a volume-level PV snapshot before recovery, pass --skip-backup on recover-region to avoid the unnecessary in-pod metadata copy.

If an environment issue prevents you from using the plugin at all, continue manually. Check the checkpoint ConfigMap to see which steps already completed, then continue with the corresponding steps in the manual path.