Troubleshoot Disaster Recovery in Confluent Manager for Apache Flink

For each reason that resources did not recover, Confluent Manager for Apache Flink® (CMF) logs a warning that names them and says what to do, as described in Verify the recovery. For most reasons, you fix the cause and restart CMF, as described in Restart CMF, which runs startup recovery again and recovers the affected resources.

The following table lists the problems whose cause or fix is less obvious, how CMF reports them, and what to do:

Symptom	Cause	What to do
metadataSync.state is FAILED, and CMF logs Decryption failed due to key mismatch or tampered data.	The encryption key is not the one that encrypted the CMF database.	Restore the original encryption key Secret, and restart CMF.
An environment or compute pool is reported with a reason that contains adopted a different key than stored, and its statements are reported as not recovered.	A Secret with the name of the key Secret of the environment or compute pool, which CMF did not create, holds another key. CMF does not overwrite it.	Delete that Secret, and restart CMF. For a suspended compute pool, delete the Secret, and then resume the compute pool.
An environment or compute pool is reported with a reason that contains owned by a different CMF instance.	Another CMF installation created the key Secret with that name in the namespace.	If the other installation no longer uses the Secret, delete it, and restart CMF.
CMF logs Not applying the changed spec of a compute pool's session cluster.	The spec of a shared compute pool’s session cluster changed while statements run on the pool, for example through an edit of the environment’s computePoolDefaults. Applying it would redeploy the cluster without taking savepoints: a stateless statement would start again without state, and other statements would not start again until you update them.	Stop the statements, update the compute pool, and resume the statements, as the warning describes. A restart of CMF applies the change when no statements block it.
CMF logs Skipping sync for resource with not this CMF instance's, and the statement or application keeps its phase.	A Kubernetes resource with the name of the statement or application exists that this CMF installation did not create, for example one of another CMF installation that uses the namespace. CMF does not overwrite it.	If nothing else uses the resource, delete it, and restart CMF. Otherwise, CMF cannot recover the statement or application under its name.
A statement is FAILED with a detail that contains no durable plan, and says that no copy of the plan is stored, or that no digest of the plan was recorded.	The plan was never copied to artifact storage, because artifact management was off, or CMF had not copied it yet.	Recreate the statement as described in Recreate a statement. It starts without its old state.
A statement is FAILED with a detail that says that the stored copy of the plan is missing.	The stored plan was deleted, cmf.artifacts.basePath does not point where the plan was stored, or the artifact storage is a copy that does not hold every plan yet.	Correct cmf.artifacts.basePath, or complete the copy, and restart CMF for the other statements. Recreate this statement as described in Recreate a statement.
A statement is FAILED with a detail that contains plan integrity check failed.	The plan ConfigMap or the stored copy of the plan does not match the plan that CMF deployed, or the ConfigMap holds no plan, and CMF has no usable copy to rebuild the ConfigMap from: they were replaced, changed, or corrupted.	Check who can write to statement-plans/ and to the ConfigMaps in the namespace. For details, see Restrict write access to stored plans. Recreate the statement as described in Recreate a statement. If the detail also says that the stored copy of the plan is missing, check the cause in the previous row too.
CMF logs Recovery left resources as they were, without a usable plan to start their jobs again.	The Flink resource of the statement exists, but its plan ConfigMap is gone or unusable, and no usable copy of the plan is stored. CMF does not reapply the resource, and its job cannot start again, for example when its pods restart.	Follow the advice in the warning, which depends on the reason that it names. If the plan is lost, recreate the statement as described in Recreate a statement, which starts it without its state.
A statement is PENDING with a detail that says that the plan store, which is the statement-plans/ prefix of artifact storage, could not be read or did not answer, that it holds no plans, or that it does not hold the identity marker .cmf-plan-store-id of this CMF installation.	CMF cannot read artifact storage, cmf.artifacts.basePath points to another location, the store is a copy that is still being made, or the marker or the plans were deleted.	Fix the access to artifact storage or the base path, or wait until the copy is complete, and restart CMF. If the marker was deleted, write the ID that the CMF log names back into it, and restart CMF. Recreate a statement whose plan was deleted as described in Recreate a statement.
A statement on a dedicated compute pool is PENDING with a detail that says that Kubernetes changes the spec that CMF applies to its Flink resource.	A mutating admission webhook changes the FlinkDeployment, or the custom resource definitions of the Flink Kubernetes Operator are older than CMF expects. CMF tries each write as a dry run first, and does not apply a spec that Kubernetes would change. If only the real write changes it, an existing resource keeps the changed spec, and a resource that CMF has just created is deleted. The detail says which, and CMF logs both specs.	Install a version of the operator that supports your CMF version, with its custom resource definitions, or change the webhook, and restart CMF.
A recreated statement stays PENDING, with the detail Recreating the Flink resource after CMF restart. or with an error that the Flink Kubernetes Operator reports. On a dedicated compute pool, a statement whose job is not submitted within 15 minutes goes to RECONCILING with the error Job Not Found.	Its job does not start, for example because an image cannot be pulled, a Secret that its pods use is missing, or its compute pool’s session cluster does not start.	Check the Flink pods and the session cluster on the replacement cluster.
A statement on a shared compute pool is RECONCILING with a detail that says that its Flink job is no longer on the session cluster of its compute pool.	The session cluster lost the job, for example because its JobManager restarted without high availability. The Flink Kubernetes Operator submits a lost stateless job again by itself, but not the others.	Update the statement, for example stop and resume it. CMF then replaces its FlinkSessionJob, and the job restores the state described in How CMF chooses the restore point. An interactive SELECT statement starts again without state, and a batch statement whose job already ran cannot start again: recreate it as described in Recreate a statement. To keep jobs across JobManager restarts, configure high availability on the compute pool, with high-availability.type and high-availability.storageDir.
A recreated statement goes to FAILING because its job cannot restore its restore point.	The files of the savepoint are gone, or the savepoint does not match the statement’s plan.	Before the statement is FAILED, update it with startFromSavepoint set to another savepoint or checkpoint and a new savepointRedeployNonce. A FAILED statement cannot be updated.
A resource is reported with the reason could not convert to a Kubernetes resource.	Its stored spec cannot be converted, for example because an application’s startFromSavepoint names a savepoint that no longer exists, or because a statement’s compute pool no longer exists.	Correct the spec with an update: CMF applies an updated resource right away. A statement whose compute pool no longer exists cannot be recovered.
Duplicate output, or exactly-once sinks that abort each other’s transactions.	The old cluster was not fenced, and its jobs still run.	Stop the old jobs, or isolate the old cluster. See Before you start recovery.

Related content

• Disaster Recovery for Confluent Manager for Apache Flink

• How Confluent Manager for Apache Flink Recovers Flink SQL Statements