# Recover Workloads onto a Replacement Cluster with Confluent Manager for Apache Flink This topic describes how to recover the workloads that Confluent Manager for Apache Flink® (CMF) manages onto a replacement cluster after the loss of a Kubernetes cluster. For what CMF recovers, see [What CMF recovers](overview.md#cmf-dr-what-is-recovered). CMF runs on the hub cluster and registers spoke clusters through kubeconfig files, as described in [Manage Multiple Kubernetes Clusters in Confluent Manager for Apache Flink](../configure/kubernetes-clusters.md#cmf-kubernetes-clusters). You cannot change the Kubernetes cluster or the namespace of an environment after you create it. To move the workloads of a lost Kubernetes cluster to a replacement cluster, you therefore keep the cluster name in CMF, and point it to the replacement cluster. The replacement can be a new cluster, or one that survived, but not a Kubernetes cluster that CMF already manages under another name. - When CMF survives and the lost cluster is a spoke cluster, you replace the kubeconfig file that CMF uses for the cluster with one for the replacement cluster, under the same name. See [Replace a lost spoke cluster](#cmf-dr-replace-spoke). - When CMF ran in the lost cluster, which is the hub cluster `default-k8s-cluster`, you install CMF on the replacement cluster, against the surviving database. See [Replace a lost hub cluster](#cmf-dr-reinstall). - When spoke clusters were lost together with the hub cluster, you also install CMF with kubeconfig files for their replacements, under the names of the lost clusters. See [Replace lost hub and spoke clusters](#cmf-dr-replace-hub-and-spokes). In each case, CMF recovers the workloads when it starts. #### IMPORTANT Fence the old cluster before CMF starts against the replacement cluster, as described in [Before you start recovery](#cmf-dr-preconditions): two copies of a job must never run at the same time, and two CMF instances must never manage the same workloads. Do not delete the resources of a lost cluster from CMF, for example with `force=true`, and do not decommission the cluster in CMF. CMF can recover only the resources that its database holds. ## Before you start recovery Make sure of the following before CMF starts against the replacement cluster. Startup recovery runs only when CMF starts, so a precondition met later takes effect only when CMF starts again. - The old cluster is fenced: its Flink jobs are stopped, or the cluster is cut off from your sources, sinks, and checkpoint storage, for example by revoking its credentials. Keep it cut off until you have deleted its Flink resources, so that its jobs cannot start again. - Checkpoint, savepoint, and artifact storage are intact, and CMF and the Flink pods on the replacement cluster can reach and authenticate to them. If they authenticate with an identity that the cluster provides, such as IAM roles for service accounts or workload identity, set up that trust for the replacement cluster. - If you recover from a copy or replica of the artifact storage, the copy is complete: it holds every stored plan, in the `statement-plans/` prefix, and the file `statement-plans/.cmf-plan-store-id`, which you copy last. With asynchronous replication, wait until it has caught up. Otherwise, startup recovery can set statements to `FAILED`, which cannot be undone. For details, see [Compiled plans of SQL statements](../configure/artifacts.md#cmf-artifacts-statement-plans). - The Flink Kubernetes Operator is installed on the replacement cluster, with its custom resource definitions and the cert-manager that it needs, and watches the namespace of each environment. Install a version that supports your CMF version, preferably the one that ran on the old cluster. See [Install and Upgrade Confluent Manager for Apache Flink](../installation/overview.md#cpf-install) and [Versions and Interoperability for Confluent Manager for Apache Flink](../installation/versions-interoperability.md#cmf-interop). - The namespace of each environment exists. CMF never creates namespaces. - CMF has the Kubernetes permissions on the replacement cluster that it had on the old one. On the hub cluster, the Helm chart grants them, unless you install it with `rbac: false`. When `watchNamespaces` lists namespaces, it grants them only in those. On a spoke cluster, grant them to the identity in the cluster’s kubeconfig file. If you install CMF with `manageEnvironmentRbac: false`, create the `ServiceAccount`, `Role`, and `RoleBinding` of each environment as described in [Pre-provision environment RBAC](../installation/helm.md#cmf-manage-environment-rbac). - The Kubernetes objects that your workloads use and that CMF does not create, listed in [Back up Kubernetes resources and configuration](prepare.md#cmf-dr-backup-k8s), exist in each namespace. - If you ran CMF 2.3 or 2.4 with shared compute pools, you deleted the duplicate statement clusters that they left, and their savepoints, as described in [Delete duplicate statement clusters left by CMF 2.3 and 2.4](../installation/upgrade-cmf.md#upgrade-cmf-stray-flinkdeployments). - Startup recovery is not turned off with `cmf.k8s.metadataSync: false`. When you replace a lost hub cluster, also make sure of the following: - The old CMF cannot start again: two CMF instances must never manage the same workloads. Revoke the credentials of the old CMF for the database, the spoke clusters, and the artifact storage. With SQLite, the old CMF has its own copy of the database, so it can start again with that copy unless you revoke its other credentials. - The CMF database is intact and reachable. If it was lost, restore it first. For details, see [Back up CMF metadata](prepare.md#cmf-dr-backup-metadata). - The CMF configuration is the same as before the disaster, in particular `cmf.artifacts` with the same `basePath`, and the encryption settings with the same key. ## Replace a lost spoke cluster Use this procedure when CMF still runs and a spoke cluster that runs your workloads is lost. CMF connects to a spoke cluster with the kubeconfig file in its kubeconfig directory that has the cluster’s name. See [Manage Multiple Kubernetes Clusters in Confluent Manager for Apache Flink](../configure/kubernetes-clusters.md#cmf-kubernetes-clusters). 1. Make sure that the preconditions in [Before you start recovery](#cmf-dr-preconditions) are met. 2. Scale CMF down to zero pods and wait for the pod to be deleted: ```bash kubectl scale deployment confluent-manager-for-apache-flink --replicas=0 -n kubectl wait --for=delete pod -l app.kubernetes.io/name=confluent-manager-for-apache-flink --timeout=60s -n ``` 3. Replace the kubeconfig file of the lost cluster with one for the replacement cluster, under the same name. For example, when the kubeconfig files are mounted from the `cmf-kubeconfigs` `Secret`, as described in [Mount kubeconfig files from a Kubernetes Secret](../configure/kubernetes-clusters.md#cmf-k8s-kubeconfig-injection), and the lost cluster is `spoke-us-west`, replace that key of the `Secret`: ```bash kubectl patch secret cmf-kubeconfigs -n --type merge \ -p "{\"data\":{\"spoke-us-west\":\"$(base64 < /local/path/to/replacement.kubeconfig | tr -d '\n')\"}}" ``` Keep the kubeconfig files of the other clusters: when CMF starts, it marks a cluster whose kubeconfig file is missing as `DISCONNECTED`. If a tool keeps the `Secret` in sync with another source, change the kubeconfig file in that source instead. 4. Scale CMF back up: ```bash kubectl scale deployment confluent-manager-for-apache-flink --replicas=1 -n ``` 5. Wait for startup recovery to finish, and verify the result as described in [Verify the recovery](#cmf-dr-verify). ### Restart CMF A restart of CMF runs startup recovery again, for example after you fix the cause of a problem. To restart CMF: ```bash kubectl rollout restart deployment/confluent-manager-for-apache-flink -n ``` With the chart’s default deployment strategy, `Recreate`, the old CMF pod stops before the new one starts, so two instances never run at the same time. Do not change this strategy with the `deploymentOverrides` Helm value. ## Replace a lost hub cluster Use this procedure when CMF ran in the lost Kubernetes cluster. It requires a CMF database that survived, or that you can restore: a JDBC database outside the lost cluster, or a backup. A SQLite database on a persistent volume of the lost cluster is lost with it, unless you restore it from a volume snapshot. Environments on the hub cluster, `default-k8s-cluster`, are recovered on the cluster where you install CMF. The workloads on spoke clusters that survived keep running, and CMF reapplies their resources. 1. Make sure that the preconditions in [Before you start recovery](#cmf-dr-preconditions) are met. If you have no record of the namespaces of your environments, you can read them from CMF: install CMF as in the following steps, and list the environments with `GET /cmf/api/v1/environments`, which CMF answers while startup recovery runs. Each environment shows its `kubernetesNamespace` and `kubernetesClusterName`. Create the missing namespaces and objects, and then restart CMF, as described in [Restart CMF](#cmf-dr-restart), so that startup recovery runs again. 2. Create the Kubernetes `Secrets` that your Helm values reference in the CMF namespace, such as the encryption key, with the same key as before, the kubeconfig files of your spoke clusters, and the credentials for the database and the artifact storage. 3. Install CMF on the replacement cluster with the same Helm values as before, as described in [Install Confluent Manager for Apache Flink with Helm](../installation/helm.md#install-cmf-helm), and pin its version with `--version`. Install the same CMF version as before if you can: CMF does not support downgrades, and a later version also upgrades CMF. If you restore a SQLite database from a volume snapshot, install CMF as described in [Restore from a snapshot](sqlite-volume-snapshots.md#cmf-dr-restore-snapshot). 4. If the address of CMF changed, update the DNS names and the clients that use CMF, such as the Confluent CLI. 5. Wait for startup recovery to finish, and verify the result as described in [Verify the recovery](#cmf-dr-verify). ## Replace lost hub and spoke clusters Use this procedure when spoke clusters were lost together with the hub cluster. Make sure that the preconditions in [Before you start recovery](#cmf-dr-preconditions) are met on each replacement cluster, and follow [Replace a lost hub cluster](#cmf-dr-reinstall). When you create the `Secrets`, use a kubeconfig file for the replacement of each lost spoke cluster, under the name of the lost cluster. ## Verify the recovery 1. Wait for startup recovery to finish: poll the system information until `metadataSync.state` is `COMPLETED`. It is `IN_PROGRESS` while startup recovery runs. A response without a `metadataSync` field means that startup recovery is turned off. ```bash curl http://cmf:8080/cmf/api/v1/system-information ``` If it is `FAILED`, CMF logs the cause at the `ERROR` level, for example with `Boot metadata sync failed` or `Reconciling pending deletions of statements failed on startup`. Fix the cause and restart CMF, as described in [Restart CMF](#cmf-dr-restart), which runs startup recovery again. To read the log of CMF: ```bash kubectl logs deployment/confluent-manager-for-apache-flink -c cmf -n ``` 2. Read the summary that CMF logs when startup recovery finishes. It starts with `Metadata recovery complete`, counts the resources of each kind that CMF processed, and says how many did not recover. For the resources that did not recover, CMF logs a warning for each kind and reason that names them and says what to do. To act on them, see [Troubleshoot Disaster Recovery in Confluent Manager for Apache Flink](troubleshoot.md#cmf-dr-troubleshoot). A session cluster that startup recovery did not update counts as recovered, so look for the warning `Not applying the changed spec of a compute pool's session cluster` too. The warning `Recovery could not restore resources; fix the cause and restart CMF to retry them` also lists the statements that this startup recovery stopped, which a restart does not start. For details, see [Statements that recovery parks](statements.md#cmf-dr-parked). 3. Check what each recreated statement restored. CMF logs one message, `Recovery recreated the missing Flink resources of statements; check the age of each restore point`, that lists each recreated statement with its restore point, for example `restored from savepoint 'orders-sp-42' (source SCHEDULE, created 2026-10-28T09:00:00Z, age PT14H), the statement's latest savepoint`, or with `recovered (stateless)` for a statement that starts without state. A statement that restores an old savepoint processes all input again since then, or skips input that is no longer retained. 4. Check the phases of your statements, and act on those that did not recover, as described in [Check your statements after recovery](statements.md#cmf-dr-statements-check). For example, the following command lists the statements of the environment `env-1` with their phases, the reasons why recovery stopped them, and their status details. The response is paginated: use the `page` parameter for more statements. ```bash curl -s "http://cmf:8080/cmf/api/v1/environments/env-1/statements?size=100" \ | jq -r '.items[] | [.metadata.name, (.status.phase // ""), (.status.traits.parkedByRecovery // ""), (.status.detail // "")] | @tsv' ``` 5. Check your applications and shared compute pools. The `status.phase` of each shared compute pool that is not suspended is `RUNNING`. If startup recovery recreated your applications, the `status.jobStatus.state` of each is `RUNNING`, and an application whose spec names no savepoint runs without its state. For details, see [Take savepoints regularly](prepare.md#cmf-dr-savepoints). If you turned off the recovery of applications, start them as described in [Recovery settings](overview.md#cmf-dr-recovery-settings). 6. If you restored the CMF database from a backup, the workloads that you stopped or deleted after the backup run again. Stop or delete them again. 7. Optionally, check the recovery metrics: - `cmf_metadata_sync_objects_total` counts the resources that startup recovery processed, by `resource` (for example `application`, `environment`, `compute-pool`, or `statement`) and `outcome`: `ok`, `skipped`, `failed`, or `parked` for a statement that an earlier recovery stopped and that was not resumed since. The `skipped` outcome includes the applications that `cmf.recovery.applications.enabled: false` leaves out. - `cmf_statements{phase="PENDING"}` falls as recreated statements start. The statements that stay `PENDING` did not start, or were not recreated. ## Related content - [Disaster Recovery for Confluent Manager for Apache Flink](overview.md#backup-restore-cmf) - [Prepare for Disaster Recovery in Confluent Manager for Apache Flink](prepare.md#cmf-dr-prepare) - [How Confluent Manager for Apache Flink Recovers Flink SQL Statements](statements.md#cmf-dr-statements) - [Troubleshoot Disaster Recovery in Confluent Manager for Apache Flink](troubleshoot.md#cmf-dr-troubleshoot) - [Manage Multiple Kubernetes Clusters in Confluent Manager for Apache Flink](../configure/kubernetes-clusters.md#cmf-kubernetes-clusters) - [Install Confluent Manager for Apache Flink with Helm](../installation/helm.md#install-cmf-helm)