Confluent Developer Marketplace Get started
  1. Home
  2. Disaster Recovery

Disaster Recovery for Confluent Manager for Apache Flink

This section describes how to protect the Apache Flink® workloads that Confluent Manager for Apache Flink (CMF) manages against the loss of a Kubernetes cluster, and how to recover them.

Disaster recovery with CMF has two parts:

  • Back up CMF metadata. The CMF database holds every environment, compute pool, application, statement, and savepoint, and nothing can be recovered without it.

  • Recover workloads. When the Kubernetes cluster that runs your Flink workloads is lost and the CMF database survives, you start CMF against a replacement cluster, and CMF recreates the Kubernetes resources of your applications, compute pools, and statements from its database.

Important

Starting with CMF 2.5.0, CMF can recover Flink SQL statements, and the session clusters of shared compute pools, after the loss of their Kubernetes cluster. It can recover a statement only if it stored a copy of the statement’s compiled plan before the disaster, which requires artifact management. Artifact management is off by default. See Prepare for Disaster Recovery in Confluent Manager for Apache Flink.

This section contains the following topics:

Your responsibilities

CMF recovers workloads from what survives the disaster. The following are your responsibilities:

  • The CMF database. Back it up so that a copy survives the loss of the cluster that runs CMF, and restore it before you recover workloads if it was lost.

  • Checkpoint, savepoint, and artifact storage. CMF stores the locations of savepoints, not their data. Keep checkpoints, savepoints, and artifacts on durable storage outside the Kubernetes cluster, such as Amazon S3, Google Cloud Storage, Azure Blob Storage, or HDFS, and make sure that a replacement cluster can reach it.

  • Starting CMF. CMF runs as a single instance and recovers workloads only when it starts. It does not fail over by itself: you start it against the surviving database.

What CMF recovers

Each time CMF starts, it runs startup recovery, also called metadata sync. Startup recovery reads the CMF database and recreates the Kubernetes resources of the workloads that CMF manages, or reapplies or keeps them if they still exist. It handles the resources in the order of the following table, so that the objects that each step needs exist before the step runs.

Resource

Kubernetes objects that CMF recreates

Notes

Applications

The FlinkDeployment of each application.

Left out when cmf.recovery.applications.enabled is false. See Recovery settings.

Environments

The ServiceAccount cmf-env-sa-<environment> with its Role and RoleBinding, and the key Secret cmf-env-key-<environment>.

The key encrypts the compiled plans of statements on compute pools that have no key of their own.

Shared compute pools

For each running pool, the key Secret cmf-pool-key-<n>-<environment>-<compute_pool>, if the pool has one, and then its session cluster FlinkDeployment.

Only pools created with CMF 2.5.0 or later have their own key. <n> is the number of characters in the name of the environment. A suspended pool gets only the Secret: it has no session cluster while it is suspended, and resuming it creates one.

Dedicated compute pools

n/a

The pool has no Kubernetes objects of its own. Each statement on it runs in its own FlinkDeployment, which is recovered with the statement.

Statements

The plan ConfigMap, and then the Flink resource: a FlinkSessionJob on a shared compute pool, or a FlinkDeployment on a dedicated one.

Only statements that are meant to be running, that is, that you have not stopped or deleted, and that have not completed or failed. CMF rebuilds a missing plan ConfigMap from the copy in artifact storage. For the savepoint that a statement’s job restores, its restore point, see How Confluent Manager for Apache Flink Recovers Flink SQL Statements.

Savepoints

n/a

CMF does not recreate FlinkStateSnapshots. It keeps the path of each completed savepoint in its database, and the Flink resources that it recreates restore a savepoint from its path in your savepoint storage. A savepoint that had not completed and whose FlinkStateSnapshot is gone cannot complete, so CMF sets it to FAILED in its database.

Note

Startup recovery generates a new spec for the session cluster of each shared compute pool from the pool and the current computePoolDefaults of its environment. If the cluster runs with another spec, for example because the defaults have changed, and statements on the pool are meant to be running, recovery does not apply the new spec. Applying it would make the Flink Kubernetes Operator redeploy the cluster without taking savepoints. Recovery leaves the cluster and its statements unchanged and logs a warning. To update the compute pool to match the defaults, stop the statements, update the pool, and resume them. For details, see Troubleshoot Disaster Recovery in Confluent Manager for Apache Flink.

Catalogs, secrets, secret mappings, detached savepoints, artifacts, and Kubernetes cluster registrations need no recovery: CMF keeps them in its database. The files of artifacts stay in artifact storage.

While startup recovery runs

Startup recovery runs in the background after CMF starts serving requests. Until it finishes, CMF rejects the following requests with HTTP 503:

  • Requests that create, change, or delete environments, compute pools, applications, statements, and the savepoints of applications and statements. Requests that detach a savepoint are accepted.

  • Requests for the results of interactive SELECT statements.

Other requests, such as reads, are unaffected. To follow its progress, see Verify the recovery.

Startup recovery does not wait for a session cluster to be ready: the Flink Kubernetes Operator starts the session jobs of a session cluster when it is up.

Recovery settings

Helm value

Default

Effect of false

cmf.k8s.metadataSync

true

Turns startup recovery off. This is an advanced setting for administrators who decide not to use startup recovery. While it is false, CMF neither recreates nor reapplies the Kubernetes resources of your workloads when it starts.

cmf.recovery.applications.enabled

true

Startup recovery recovers everything except applications, and logs how many it left out.

With the default, a recovered application restores the savepoint that its spec names, which can be old, or starts without state. That suits stateless applications.

To start stateful applications from savepoints that you choose during a failover:

  1. Set cmf.recovery.applications.enabled to false before you start CMF.

  2. After startup recovery has finished, start each stateful application from a savepoint, as described in Start a job from a savepoint.

  3. Set the value back to true. CMF restarts, and startup recovery recovers the applications that you did not start, as with the default.

The value applies to every start of CMF. While it is false, startup recovery does not recreate the applications of a lost cluster, and a restart does not reapply their specs.

Related content