# Disaster Recovery for Confluent Manager for Apache Flink This section describes how to protect the Apache Flink® workloads that Confluent Manager for Apache Flink (CMF) manages against the loss of a Kubernetes cluster, and how to recover them. Disaster recovery with CMF has two parts: - Back up CMF metadata. The CMF database holds every environment, compute pool, application, statement, and savepoint, and nothing can be recovered without it. - Recover workloads. When the Kubernetes cluster that runs your Flink workloads is lost and the CMF database survives, you start CMF against a replacement cluster, and CMF recreates the Kubernetes resources of your applications, compute pools, and statements from its database. #### IMPORTANT Starting with CMF 2.5.0, CMF can recover Flink SQL statements, and the session clusters of shared compute pools, after the loss of their Kubernetes cluster. It can recover a statement only if it stored a copy of the statement’s compiled plan before the disaster, which requires artifact management. Artifact management is off by default. See [Prepare for Disaster Recovery in Confluent Manager for Apache Flink](prepare.md#cmf-dr-prepare). This section contains the following topics: - [Prepare for Disaster Recovery in Confluent Manager for Apache Flink](prepare.md#cmf-dr-prepare): Describes what to do before a disaster: back up the CMF metadata, store the compiled plans of statements durably, keep checkpoints and savepoints on durable storage, and take savepoints. - [Back Up and Restore SQLite Metadata of Confluent Manager for Apache Flink with Volume Snapshots](sqlite-volume-snapshots.md#cmf-dr-volume-snapshots): Describes how to back up and restore the default SQLite database of CMF with Kubernetes volume snapshots. - [Recover Workloads onto a Replacement Cluster with Confluent Manager for Apache Flink](recover.md#cmf-dr-recover): Describes how to recover workloads onto a replacement cluster after the loss of a spoke cluster or of the cluster that runs CMF, and how to verify the result. - [How Confluent Manager for Apache Flink Recovers Flink SQL Statements](statements.md#cmf-dr-statements): Describes how to check your Flink SQL statements after recovery, how CMF recovers each statement and chooses the state that a recreated statement starts from, and how to resume or recreate a statement that recovery stopped. - [Troubleshoot Disaster Recovery in Confluent Manager for Apache Flink](troubleshoot.md#cmf-dr-troubleshoot): Lists the problems that keep workloads from being recovered, how CMF reports them, and what to do. ## Your responsibilities CMF recovers workloads from what survives the disaster. The following are your responsibilities: - The CMF database. Back it up so that a copy survives the loss of the cluster that runs CMF, and restore it before you recover workloads if it was lost. - Checkpoint, savepoint, and artifact storage. CMF stores the locations of savepoints, not their data. Keep checkpoints, savepoints, and artifacts on durable storage outside the Kubernetes cluster, such as Amazon S3, Google Cloud Storage, Azure Blob Storage, or HDFS, and make sure that a replacement cluster can reach it. - Starting CMF. CMF runs as a single instance and recovers workloads only when it starts. It does not fail over by itself: you start it against the surviving database. ## What CMF recovers Each time CMF starts, it runs startup recovery, also called metadata sync. Startup recovery reads the CMF database and recreates the Kubernetes resources of the workloads that CMF manages, or reapplies or keeps them if they still exist. It handles the resources in the order of the following table, so that the objects that each step needs exist before the step runs. | Resource | Kubernetes objects that CMF recreates | Notes | |-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Applications | The `FlinkDeployment` of each application. | Left out when `cmf.recovery.applications.enabled` is `false`. See
[Recovery settings](#cmf-dr-recovery-settings). | | Environments | The `ServiceAccount` `cmf-env-sa-` with its `Role` and
`RoleBinding`, and the key `Secret` `cmf-env-key-`. | The key encrypts the compiled plans of statements on compute pools that have no key of
their own. | | Shared compute pools | For each running pool, the key `Secret` `cmf-pool-key---`,
if the pool has one, and then its session cluster `FlinkDeployment`. | Only pools created with CMF 2.5.0 or later have their own key. `` is the number of
characters in the name of the environment. A suspended pool gets only the `Secret`: it
has no session cluster while it is suspended, and resuming it creates one. | | Dedicated compute pools | n/a | The pool has no Kubernetes objects of its own. Each statement on it runs in its own
`FlinkDeployment`, which is recovered with the statement. | | Statements | The plan `ConfigMap`, and then the Flink resource: a `FlinkSessionJob` on a shared
compute pool, or a `FlinkDeployment` on a dedicated one. | Only statements that are meant to be running, that is, that you have not stopped or
deleted, and that have not completed or failed. CMF rebuilds a missing plan
`ConfigMap` from the copy in artifact storage. For the savepoint that a statement’s job
restores, its restore point, see [How Confluent Manager for Apache Flink Recovers Flink SQL Statements](statements.md#cmf-dr-statements). | | Savepoints | n/a | CMF does not recreate `FlinkStateSnapshots`. It keeps the path of each completed
savepoint in its database, and the Flink resources that it recreates restore a savepoint from
its path in your savepoint storage. A savepoint that had not completed and whose
`FlinkStateSnapshot` is gone cannot complete, so CMF sets it to `FAILED` in its
database. | #### NOTE Startup recovery generates a new spec for the session cluster of each shared compute pool from the pool and the current `computePoolDefaults` of its environment. If the cluster runs with another spec, for example because the defaults have changed, and statements on the pool are meant to be running, recovery does not apply the new spec. Applying it would make the Flink Kubernetes Operator redeploy the cluster without taking savepoints. Recovery leaves the cluster and its statements unchanged and logs a warning. To update the compute pool to match the defaults, stop the statements, update the pool, and resume them. For details, see [Troubleshoot Disaster Recovery in Confluent Manager for Apache Flink](troubleshoot.md#cmf-dr-troubleshoot). Catalogs, secrets, secret mappings, detached savepoints, artifacts, and Kubernetes cluster registrations need no recovery: CMF keeps them in its database. The files of artifacts stay in artifact storage. ### While startup recovery runs Startup recovery runs in the background after CMF starts serving requests. Until it finishes, CMF rejects the following requests with HTTP 503: - Requests that create, change, or delete environments, compute pools, applications, statements, and the savepoints of applications and statements. Requests that detach a savepoint are accepted. - Requests for the results of interactive `SELECT` statements. Other requests, such as reads, are unaffected. To follow its progress, see [Verify the recovery](recover.md#cmf-dr-verify). Startup recovery does not wait for a session cluster to be ready: the Flink Kubernetes Operator starts the session jobs of a session cluster when it is up. ### Recovery settings | Helm value | Default | Effect of `false` | |-------------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `cmf.k8s.metadataSync` | `true` | Turns startup recovery off. This is an advanced setting for administrators who decide not
to use startup recovery. While it is `false`, CMF neither recreates nor reapplies the
Kubernetes resources of your workloads when it starts. | | `cmf.recovery.applications.enabled` | `true` | Startup recovery recovers everything except applications, and logs how many it left out. | With the default, a recovered application restores the savepoint that its spec names, which can be old, or starts without state. That suits stateless applications. To start stateful applications from savepoints that you choose during a failover: 1. Set `cmf.recovery.applications.enabled` to `false` before you start CMF. 2. After startup recovery has finished, start each stateful application from a savepoint, as described in [Start a job from a savepoint](../jobs/savepoints.md#cmf-savepoints-start). 3. Set the value back to `true`. CMF restarts, and startup recovery recovers the applications that you did not start, as with the default. The value applies to every start of CMF. While it is `false`, startup recovery does not recreate the applications of a lost cluster, and a restart does not reapply their specs. ## Related content - [Configure Storage for Confluent Manager for Apache Flink](../installation/storage.md#cmf-storage) - [Data Encryption in Confluent Manager for Apache Flink](../installation/encryption.md#cmf-encrypt) - [Manage Artifacts in Confluent Manager for Apache Flink](../configure/artifacts.md#cmf-artifacts) - [Manage Multiple Kubernetes Clusters in Confluent Manager for Apache Flink](../configure/kubernetes-clusters.md#cmf-kubernetes-clusters) - [Savepoint Management in Confluent Manager for Apache Flink](../jobs/savepoints.md#cmf-savepoints) - [Checkpointing with Confluent Manager for Apache Flink](../jobs/configure/checkpointing.md#cmf-checkpointing) - [Manage Statements in Confluent Manager for Apache Flink](../jobs/sql-statements/manage-statements.md#flink-sql-statements-manage)