Recover Workloads onto a Replacement Cluster with Confluent Manager for Apache Flink

This topic describes how to recover the workloads that Confluent Manager for Apache Flink® (CMF) manages onto a replacement cluster after the loss of a Kubernetes cluster. For what CMF recovers, see What CMF recovers.

CMF runs on the hub cluster and registers spoke clusters through kubeconfig files, as described in Manage Multiple Kubernetes Clusters in Confluent Manager for Apache Flink.

You cannot change the Kubernetes cluster or the namespace of an environment after you create it. To move the workloads of a lost Kubernetes cluster to a replacement cluster, you therefore keep the cluster name in CMF, and point it to the replacement cluster. The replacement can be a new cluster, or one that survived, but not a Kubernetes cluster that CMF already manages under another name.

When CMF survives and the lost cluster is a spoke cluster, you replace the kubeconfig file that CMF uses for the cluster with one for the replacement cluster, under the same name. See Replace a lost spoke cluster.
When CMF ran in the lost cluster, which is the hub cluster default-k8s-cluster, you install CMF on the replacement cluster, against the surviving database. See Replace a lost hub cluster.
When spoke clusters were lost together with the hub cluster, you also install CMF with kubeconfig files for their replacements, under the names of the lost clusters. See Replace lost hub and spoke clusters.

In each case, CMF recovers the workloads when it starts.

Important

Fence the old cluster before CMF starts against the replacement cluster, as described in Before you start recovery: two copies of a job must never run at the same time, and two CMF instances must never manage the same workloads. Do not delete the resources of a lost cluster from CMF, for example with force=true, and do not decommission the cluster in CMF. CMF can recover only the resources that its database holds.

Before you start recovery

Make sure of the following before CMF starts against the replacement cluster. Startup recovery runs only when CMF starts, so a precondition met later takes effect only when CMF starts again.

The old cluster is fenced: its Flink jobs are stopped, or the cluster is cut off from your sources, sinks, and checkpoint storage, for example by revoking its credentials. Keep it cut off until you have deleted its Flink resources, so that its jobs cannot start again.
Checkpoint, savepoint, and artifact storage are intact, and CMF and the Flink pods on the replacement cluster can reach and authenticate to them. If they authenticate with an identity that the cluster provides, such as IAM roles for service accounts or workload identity, set up that trust for the replacement cluster.
If you recover from a copy or replica of the artifact storage, the copy is complete: it holds every stored plan, in the statement-plans/ prefix, and the file statement-plans/.cmf-plan-store-id, which you copy last. With asynchronous replication, wait until it has caught up. Otherwise, startup recovery can set statements to FAILED, which cannot be undone. For details, see Compiled plans of SQL statements.
The Flink Kubernetes Operator is installed on the replacement cluster, with its custom resource definitions and the cert-manager that it needs, and watches the namespace of each environment. Install a version that supports your CMF version, preferably the one that ran on the old cluster. See Install and Upgrade Confluent Manager for Apache Flink and Versions and Interoperability for Confluent Manager for Apache Flink.
The namespace of each environment exists. CMF never creates namespaces.
CMF has the Kubernetes permissions on the replacement cluster that it had on the old one. On the hub cluster, the Helm chart grants them, unless you install it with rbac: false. When watchNamespaces lists namespaces, it grants them only in those. On a spoke cluster, grant them to the identity in the cluster’s kubeconfig file. If you install CMF with manageEnvironmentRbac: false, create the ServiceAccount, Role, and RoleBinding of each environment as described in Pre-provision environment RBAC.
The Kubernetes objects that your workloads use and that CMF does not create, listed in Back up Kubernetes resources and configuration, exist in each namespace.
If you ran CMF 2.3 or 2.4 with shared compute pools, you deleted the duplicate statement clusters that they left, and their savepoints, as described in Delete duplicate statement clusters left by CMF 2.3 and 2.4.
Startup recovery is not turned off with cmf.k8s.metadataSync: false.

When you replace a lost hub cluster, also make sure of the following:

The old CMF cannot start again: two CMF instances must never manage the same workloads. Revoke the credentials of the old CMF for the database, the spoke clusters, and the artifact storage. With SQLite, the old CMF has its own copy of the database, so it can start again with that copy unless you revoke its other credentials.
The CMF database is intact and reachable. If it was lost, restore it first. For details, see Back up CMF metadata.
The CMF configuration is the same as before the disaster, in particular cmf.artifacts with the same basePath, and the encryption settings with the same key.

Replace a lost spoke cluster

Use this procedure when CMF still runs and a spoke cluster that runs your workloads is lost. CMF connects to a spoke cluster with the kubeconfig file in its kubeconfig directory that has the cluster’s name. See Manage Multiple Kubernetes Clusters in Confluent Manager for Apache Flink.

Make sure that the preconditions in Before you start recovery are met.

Scale CMF down to zero pods and wait for the pod to be deleted:

kubectl scale deployment confluent-manager-for-apache-flink --replicas=0 -n <cmf_namespace>
kubectl wait --for=delete pod -l app.kubernetes.io/name=confluent-manager-for-apache-flink --timeout=60s -n <cmf_namespace>

Replace the kubeconfig file of the lost cluster with one for the replacement cluster, under the same name. For example, when the kubeconfig files are mounted from the cmf-kubeconfigs Secret, as described in Mount kubeconfig files from a Kubernetes Secret, and the lost cluster is spoke-us-west, replace that key of the Secret:
```
kubectl patch secret cmf-kubeconfigs -n <cmf_namespace> --type merge \
  -p "{\"data\":{\"spoke-us-west\":\"$(base64 < /local/path/to/replacement.kubeconfig | tr -d '\n')\"}}"
```
Keep the kubeconfig files of the other clusters: when CMF starts, it marks a cluster whose kubeconfig file is missing as DISCONNECTED. If a tool keeps the Secret in sync with another source, change the kubeconfig file in that source instead.

Scale CMF back up:

kubectl scale deployment confluent-manager-for-apache-flink --replicas=1 -n <cmf_namespace>

Wait for startup recovery to finish, and verify the result as described in Verify the recovery.

Restart CMF

A restart of CMF runs startup recovery again, for example after you fix the cause of a problem. To restart CMF:

kubectl rollout restart deployment/confluent-manager-for-apache-flink -n <cmf_namespace>

With the chart’s default deployment strategy, Recreate, the old CMF pod stops before the new one starts, so two instances never run at the same time. Do not change this strategy with the deploymentOverrides Helm value.

Replace a lost hub cluster

Use this procedure when CMF ran in the lost Kubernetes cluster. It requires a CMF database that survived, or that you can restore: a JDBC database outside the lost cluster, or a backup. A SQLite database on a persistent volume of the lost cluster is lost with it, unless you restore it from a volume snapshot.

Environments on the hub cluster, default-k8s-cluster, are recovered on the cluster where you install CMF. The workloads on spoke clusters that survived keep running, and CMF reapplies their resources.

Make sure that the preconditions in Before you start recovery are met. If you have no record of the namespaces of your environments, you can read them from CMF: install CMF as in the following steps, and list the environments with GET /cmf/api/v1/environments, which CMF answers while startup recovery runs. Each environment shows its kubernetesNamespace and kubernetesClusterName. Create the missing namespaces and objects, and then restart CMF, as described in Restart CMF, so that startup recovery runs again.
Create the Kubernetes Secrets that your Helm values reference in the CMF namespace, such as the encryption key, with the same key as before, the kubeconfig files of your spoke clusters, and the credentials for the database and the artifact storage.
Install CMF on the replacement cluster with the same Helm values as before, as described in Install Confluent Manager for Apache Flink with Helm, and pin its version with --version. Install the same CMF version as before if you can: CMF does not support downgrades, and a later version also upgrades CMF. If you restore a SQLite database from a volume snapshot, install CMF as described in Restore from a snapshot.
If the address of CMF changed, update the DNS names and the clients that use CMF, such as the Confluent CLI.
Wait for startup recovery to finish, and verify the result as described in Verify the recovery.

Replace lost hub and spoke clusters

Use this procedure when spoke clusters were lost together with the hub cluster. Make sure that the preconditions in Before you start recovery are met on each replacement cluster, and follow Replace a lost hub cluster. When you create the Secrets, use a kubeconfig file for the replacement of each lost spoke cluster, under the name of the lost cluster.

Verify the recovery

Wait for startup recovery to finish: poll the system information until metadataSync.state is COMPLETED. It is IN_PROGRESS while startup recovery runs. A response without a metadataSync field means that startup recovery is turned off.
```
curl http://cmf:8080/cmf/api/v1/system-information
```
If it is FAILED, CMF logs the cause at the ERROR level, for example with Boot metadata sync failed or Reconciling pending deletions of statements failed on startup. Fix the cause and restart CMF, as described in Restart CMF, which runs startup recovery again. To read the log of CMF:
```
kubectl logs deployment/confluent-manager-for-apache-flink -c cmf -n <cmf_namespace>
```
Read the summary that CMF logs when startup recovery finishes. It starts with Metadata recovery complete, counts the resources of each kind that CMF processed, and says how many did not recover. For the resources that did not recover, CMF logs a warning for each kind and reason that names them and says what to do. To act on them, see Troubleshoot Disaster Recovery in Confluent Manager for Apache Flink. A session cluster that startup recovery did not update counts as recovered, so look for the warning Not applying the changed spec of a compute pool's session cluster too.

The warning Recovery could not restore resources; fix the cause and restart CMF to retry them also lists the statements that this startup recovery stopped, which a restart does not start. For details, see Statements that recovery parks.
Check what each recreated statement restored. CMF logs one message, Recovery recreated the missing Flink resources of statements; check the age of each restore point, that lists each recreated statement with its restore point, for example restored from savepoint 'orders-sp-42' (source SCHEDULE, created 2026-10-28T09:00:00Z, age PT14H), the statement's latest savepoint, or with recovered (stateless) for a statement that starts without state. A statement that restores an old savepoint processes all input again since then, or skips input that is no longer retained.
Check the phases of your statements, and act on those that did not recover, as described in Check your statements after recovery. For example, the following command lists the statements of the environment env-1 with their phases, the reasons why recovery stopped them, and their status details. The response is paginated: use the page parameter for more statements.
```
curl -s "http://cmf:8080/cmf/api/v1/environments/env-1/statements?size=100" \
  | jq -r '.items[] | [.metadata.name, (.status.phase // ""), (.status.traits.parkedByRecovery // ""), (.status.detail // "")] | @tsv'
```
Check your applications and shared compute pools. The status.phase of each shared compute pool that is not suspended is RUNNING. If startup recovery recreated your applications, the status.jobStatus.state of each is RUNNING, and an application whose spec names no savepoint runs without its state. For details, see Take savepoints regularly. If you turned off the recovery of applications, start them as described in Recovery settings.
If you restored the CMF database from a backup, the workloads that you stopped or deleted after the backup run again. Stop or delete them again.
Optionally, check the recovery metrics:
- cmf_metadata_sync_objects_total counts the resources that startup recovery processed, by resource (for example application, environment, compute-pool, or statement) and outcome: ok, skipped, failed, or parked for a statement that an earlier recovery stopped and that was not resumed since. The skipped outcome includes the applications that cmf.recovery.applications.enabled: false leaves out.
- cmf_statements{phase="PENDING"} falls as recreated statements start. The statements that stay PENDING did not start, or were not recreated.