# Troubleshoot Disaster Recovery in Confluent Manager for Apache Flink For each reason that resources did not recover, Confluent Manager for Apache Flink® (CMF) logs a warning that names them and says what to do, as described in [Verify the recovery](recover.md#cmf-dr-verify). For most reasons, you fix the cause and restart CMF, as described in [Restart CMF](recover.md#cmf-dr-restart), which runs startup recovery again and recovers the affected resources. The following table lists the problems whose cause or fix is less obvious, how CMF reports them, and what to do: | Symptom | Cause | What to do | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `metadataSync.state` is `FAILED`, and CMF logs `Decryption failed due to key
mismatch or tampered data`. | The encryption key is not the one that encrypted the CMF database. | Restore the original encryption key `Secret`, and restart CMF. | | An environment or compute pool is reported with a reason that contains `adopted a
different key than stored`, and its statements are reported as not recovered. | A `Secret` with the name of the key `Secret` of the environment or compute pool, which
CMF did not create, holds another key. CMF does not overwrite it. | Delete that `Secret`, and restart CMF. For a suspended compute pool, delete the
`Secret`, and then resume the compute pool. | | An environment or compute pool is reported with a reason that contains `owned by a
different CMF instance`. | Another CMF installation created the key `Secret` with that name in the namespace. | If the other installation no longer uses the `Secret`, delete it, and restart CMF. | | CMF logs `Not applying the changed spec of a compute pool's session cluster`. | The spec of a shared compute pool’s session cluster changed while statements run on the
pool, for example through an edit of the environment’s `computePoolDefaults`. Applying it
would redeploy the cluster without taking savepoints: a `stateless` statement would start
again without state, and other statements would not start again until you update them. | Stop the statements, update the compute pool, and resume the statements, as the warning
describes. A restart of CMF applies the change when no statements block it. | | CMF logs `Skipping sync for resource` with `not this CMF instance's`, and the
statement or application keeps its phase. | A Kubernetes resource with the name of the statement or application exists that this
CMF installation did not create, for example one of another CMF installation that
uses the namespace. CMF does not overwrite it. | If nothing else uses the resource, delete it, and restart CMF. Otherwise, CMF cannot
recover the statement or application under its name. | | A statement is `FAILED` with a detail that contains `no durable plan`, and says that no
copy of the plan is stored, or that no digest of the plan was recorded. | The plan was never copied to artifact storage, because artifact management was off, or CMF
had not copied it yet. | Recreate the statement as described in [Recreate a statement](statements.md#cmf-dr-recreate). It starts without its old
state. | | A statement is `FAILED` with a detail that says that the stored copy of the plan is
missing. | The stored plan was deleted, `cmf.artifacts.basePath` does not point where the plan was
stored, or the artifact storage is a copy that does not hold every plan yet. | Correct `cmf.artifacts.basePath`, or complete the copy, and restart CMF for the other
statements. Recreate this statement as described in [Recreate a statement](statements.md#cmf-dr-recreate). | | A statement is `FAILED` with a detail that contains `plan integrity check failed`. | The plan `ConfigMap` or the stored copy of the plan does not match the plan that CMF
deployed, or the `ConfigMap` holds no plan, and CMF has no usable copy to rebuild the
`ConfigMap` from: they were replaced, changed, or corrupted. | Check who can write to `statement-plans/` and to the `ConfigMaps` in the namespace.
For details, see [Restrict write access to stored plans](../configure/artifacts.md#cmf-artifacts-plan-write-access). Recreate the statement as
described in [Recreate a statement](statements.md#cmf-dr-recreate). If the detail also says that the stored copy of the
plan is missing, check the cause in the previous row too. | | CMF logs `Recovery left resources as they were, without a usable plan to start their
jobs again`. | The Flink resource of the statement exists, but its plan `ConfigMap` is gone or unusable,
and no usable copy of the plan is stored. CMF does not reapply the resource, and its job
cannot start again, for example when its pods restart. | Follow the advice in the warning, which depends on the reason that it names. If the plan is
lost, recreate the statement as described in [Recreate a statement](statements.md#cmf-dr-recreate), which starts it
without its state. | | A statement is `PENDING` with a detail that says that the plan store, which is the
`statement-plans/` prefix of artifact storage, could not be read or did not answer, that
it holds no plans, or that it does not hold the identity marker `.cmf-plan-store-id` of
this CMF installation. | CMF cannot read artifact storage, `cmf.artifacts.basePath` points to another location,
the store is a copy that is still being made, or the marker or the plans were deleted. | Fix the access to artifact storage or the base path, or wait until the copy is complete,
and restart CMF. If the marker was deleted, write the ID that the CMF log names back
into it, and restart CMF. Recreate a statement whose plan was deleted as described in
[Recreate a statement](statements.md#cmf-dr-recreate). | | A statement on a dedicated compute pool is `PENDING` with a detail that says that
Kubernetes changes the spec that CMF applies to its Flink resource. | A mutating admission webhook changes the `FlinkDeployment`, or the custom resource
definitions of the Flink Kubernetes Operator are older than CMF expects. CMF tries each
write as a dry run first, and does not apply a spec that Kubernetes would change. If only
the real write changes it, an existing resource keeps the changed spec, and a resource that
CMF has just created is deleted. The detail says which, and CMF logs both specs. | Install a version of the operator that supports your CMF version, with its custom
resource definitions, or change the webhook, and restart CMF. | | A recreated statement stays `PENDING`, with the detail `Recreating the Flink resource
after CMF restart.` or with an error that the Flink Kubernetes Operator reports. On a
dedicated compute pool, a statement whose job is not submitted within 15 minutes goes to
`RECONCILING` with the error `Job Not Found`. | Its job does not start, for example because an image cannot be pulled, a `Secret` that
its pods use is missing, or its compute pool’s session cluster does not start. | Check the Flink pods and the session cluster on the replacement cluster. | | A statement on a shared compute pool is `RECONCILING` with a detail that says that its
Flink job is no longer on the session cluster of its compute pool. | The session cluster lost the job, for example because its JobManager restarted without
high availability. The Flink Kubernetes Operator submits a lost `stateless` job again by
itself, but not the others. | Update the statement, for example stop and resume it. CMF then replaces its
`FlinkSessionJob`, and the job restores the state described in
[How CMF chooses the restore point](statements.md#cmf-dr-restore-point). An interactive `SELECT` statement starts again without state,
and a batch statement whose job already ran cannot start again: recreate it as described
in [Recreate a statement](statements.md#cmf-dr-recreate). To keep jobs across JobManager restarts, configure high
availability on the compute pool, with `high-availability.type` and
`high-availability.storageDir`. | | A recreated statement goes to `FAILING` because its job cannot restore its restore
point. | The files of the savepoint are gone, or the savepoint does not match the statement’s plan. | Before the statement is `FAILED`, update it with `startFromSavepoint` set to another
savepoint or checkpoint and a new `savepointRedeployNonce`. A `FAILED` statement cannot
be updated. | | A resource is reported with the reason `could not convert to a Kubernetes resource`. | Its stored spec cannot be converted, for example because an application’s
`startFromSavepoint` names a savepoint that no longer exists, or because a statement’s
compute pool no longer exists. | Correct the spec with an update: CMF applies an updated resource right away. A statement
whose compute pool no longer exists cannot be recovered. | | Duplicate output, or exactly-once sinks that abort each other’s transactions. | The old cluster was not fenced, and its jobs still run. | Stop the old jobs, or isolate the old cluster. See [Before you start recovery](recover.md#cmf-dr-preconditions). | ## Related content - [Disaster Recovery for Confluent Manager for Apache Flink](overview.md#backup-restore-cmf) - [How Confluent Manager for Apache Flink Recovers Flink SQL Statements](statements.md#cmf-dr-statements)