Upgrade Confluent Manager for Apache Flink
Flink applications are deployed in Confluent Platform with Confluent Manager for Apache Flink®, which is a central management component that enables users to securely manage a fleet of Flink applications across multiple environments. Use these instructions to upgrade earlier versions of Confluent Manager for Apache Flink (CMF) to a later version. To install CMF, see Install Confluent Manager for Apache Flink with Helm.
Warning
Downgrades are not supported by CMF. Before upgrading, ensure you have tested the new version in a non-production environment.
Important
If you upgrade to CMF 2.5 from an earlier version, first complete the checks in Upgrade to CMF 2.5.
If you ran CMF 2.3 or 2.4 with shared compute pools, read Delete duplicate statement clusters left by CMF 2.3 and 2.4 before you start: its first step must be done while CMF still runs.
Follow these steps to upgrade CMF:
Scale down CMF to zero (0) pods and wait for the pod to be deleted. The following command shows an example of how to scale down CMF:
kubectl scale deployment confluent-manager-for-apache-flink --replicas=0 -n <namespace> kubectl wait --for=delete pod -l app.kubernetes.io/name=confluent-manager-for-apache-flink --timeout=60s -n <namespace>
Uninstall the existing Flink Kubernetes Operator:
helm uninstall cp-flink-kubernetes-operator -n <namespace>
Ensure certificate manager is installed, or use the following command to install it:
kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.18.2/cert-manager.yaml kubectl wait --for=condition=Available deployment -n cert-manager cert-manager-webhook --timeout=180s
Find the version Confluent Platform for Apache Flink Kubernetes operator you want to install. For a list of available versions, see the Versions and Interoperability for Confluent Manager for Apache Flink.
helm repo add confluentinc https://packages.confluent.io/helm helm repo update confluentinc helm upgrade --install --version "<desired_version>" cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator
Upgrade CMF to the desired version using the Helm command:
helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink --namespace <namespace> ...
Important
If the CMF database is file system based, using helm uninstall cmf will
lead to the loss of the database because the Persistent Volume Claim (PVC) is managed by CMF.
Delete duplicate statement clusters left by CMF 2.3 and 2.4
Each time they started, CMF 2.3.0 to 2.3.2 and 2.4.0 to 2.4.2 created or replaced a standalone
FlinkDeployment with the name of each running, pending, failing, or stopped statement on a
shared compute pool. Unless the statement was stopped the last time one of these versions started,
this cluster can run a second copy of the statement’s job, which reads the same sources and
writes to the same sinks. Stopping, resuming, or deleting the statement does not stop it.
CMF can also take the status and errors of such a cluster for those of the statement, and
attach to the statement the savepoints that the Flink Kubernetes Operator takes of the cluster.
Recovery in CMF 2.5.0 can then restore those savepoints.
Later 2.3 and 2.4 versions create such a cluster only when a read of a compute pool from the
database fails once while CMF starts. They then log Unable to determine compute pool type for
statement. CMF 2.5.0 never creates them, and no version cleans them up. If you ran CMF 2.3
or 2.4 with shared compute pools, delete them as soon as possible. While you still run 2.3.0 to
2.3.2 or 2.4.0 to 2.4.2, which create them again each time they start, delete them as part of the
upgrade to a version that does not. Those are 2.3.3 and later 2.3 versions, 2.4.3 and later 2.4
versions, and 2.5.0 and later. On such a version, you can also delete them on their own:
Right before you scale CMF down, while it still runs, note the namespace and the Kubernetes cluster of each environment (
kubernetesNamespaceandkubernetesClusterName), and the names of the environment’s statements on dedicated compute pools, in any phase. You need this information in step 4, while CMF is down. A statement’s compute pool is in itsspec.computePoolName, and thespec.typeof a compute pool isDEDICATEDorSHARED. Do not create statements until you have finished step 4.Scale CMF down to zero pods, as in step 1 of the upgrade: while it runs, CMF can take the status of a cluster that you delete for that of the statement. Delete the clusters before you uninstall the Flink Kubernetes Operator: without the operator, the deletion of a
FlinkDeploymentdoes not complete, and its job keeps running.On the Kubernetes cluster of each environment, list the
FlinkDeploymentsthat CMF created for statements in the environment’s namespace, with the environment of each in theENVIDcolumn. Include the namespaces of environments that you deleted: deleting an environment does not delete these clusters.kubectl get flinkdeployment -n <namespace> -l confluent.io/cmfResource=Statement -L confluent.io/envId
Delete each listed
FlinkDeploymentwhoseENVIDis the name of one of your environments, or of one that you deleted, and whose name is not that of a statement on a dedicated compute pool in that environment:kubectl delete flinkdeployment <name> -n <namespace>
Note the following:
A statement on a dedicated compute pool runs as the
FlinkDeploymentwith its name, so do not delete those.A statement whose compute pool no longer exists is
FAILEDorCOMPLETEDand cannot run again, so you can delete itsFlinkDeploymenttoo.If an application has the name of a statement on a shared compute pool, the listed
FlinkDeploymentof that name replaced the application’s own. When CMF starts again, it recreates the application, from the savepoint that its spec names, or without state.Leave the other listed
FlinkDeployments, which belong to other CMF installations that use the namespace. If such an installation has an environment with the name of one of yours, these labels do not tell itsFlinkDeploymentsfrom yours. Check with its administrators before you delete any.The
FlinkDeploymentsof applications and the session clusters of compute pools have another value ofconfluent.io/cmfResource, and are not listed.
Continue with step 2 of the upgrade, or, if you delete the clusters on their own, scale CMF back up.
After CMF runs again, delete the savepoints that CMF attached from the deleted clusters. For each statement on a shared compute pool, list the savepoints that were taken of a
FlinkDeployment. The response is paginated: use thepageparameter for more savepoints.curl -s "http://cmf:8080/cmf/api/v1/environments/<environment>/statements/<statement>/savepoints?size=100" \ | jq -r '.items[] | select(.metadata.labels["job-reference.kind"] == "FlinkDeployment") | .metadata.name'
Delete each listed savepoint with
force=true:curl -X DELETE "http://cmf:8080/cmf/api/v1/environments/<environment>/statements/<statement>/savepoints/<savepoint>?force=true"
Then delete the
FlinkStateSnapshotsthat remain from eachFlinkDeploymentthat you deleted in step 4, so that CMF does not attach them to a statement or application of the same name later:kubectl delete flinkstatesnapshot -n <namespace> -l job-reference.kind=FlinkDeployment,job-reference.name=<name>
Upgrade to CMF 2.5
Starting with version 2.5, CMF compiles SQL statements in shared compilers, which run as two extra containers in the CMF pod, and statements that use artifacts compile only on compile workers. For details, see Configure SQL Compilers and Compile Workers for Confluent Manager for Apache Flink.
CMF 2.5.0 can recover Flink SQL statements after the loss of their Kubernetes cluster from
copies of their compiled plans, which it stores in artifact storage. To protect the statements
that exist when you upgrade, enable artifact management, and wait until CMF has copied their
plans while their cluster still exists. Then take a savepoint of each statement that has a
startFromSavepoint, as described in Statements with a startFromSavepoint. For all the
steps, see Prepare for Disaster Recovery in Confluent Manager for Apache Flink.
If you return to an earlier version by restoring a backup of the CMF database that you took
before the upgrade, that version neither uses nor deletes the plans that CMF 2.5.0 stored under
statement-plans/ in artifact storage, and you can delete them.
Set up compile workers
If your SQL statements use artifacts, such as user-defined functions or custom connectors, with
cmf.artifacts.enabled and cmf.sql.environmentCatalog.enabled both true, then after the
upgrade every statement of an environment that has an artifact compiles on a compile worker,
including statements that use no artifact. The chart adds one compile worker by default. Before
you upgrade, set up the compile workers:
Copy the environment variables and volumes from the top-level
extraEnvandmountedVolumesthat your catalogs and connectors use intocmf.statementService.workers.extraEnvandcmf.statementService.workers.mountedVolumes. Mark each volume mountreadOnly: true, and leave out JVM option variables such asJAVA_TOOL_OPTIONS. Otherwise the chart refuses the values. See Give the workers what catalogs and connectors need.Copy the JVM options that catalogs need, such as a trust store, from the top-level
jvmArgstocmf.statementService.workers.compilerJvmArgs, and keep a heap size there, such as-Xmx1g.Workers do not get the cloud identity of the CMF ServiceAccount. A catalog that uses it needs credentials in its options, or a ServiceAccount for the workers.
If you set the top-level
nodeSelector,tolerations,affinity,topologySpreadConstraints,priorityClassName,podLabels, orpodAnnotations, copy what the worker pod needs intoworkerDeploymentOverrides; see Compile worker settings. The worker pod gets none of them. If your nodes are all tainted, or admission policies require pod labels, the worker pod staysPending,helm upgrade --waittimes out, and--atomicrolls back the upgrade. If no SQL statement uses an artifact, you can instead setcmf.statementService.workers.count: 0.Set
cmf.statementService.workers.countfor the load; see Size the compile workers.If you render the chart with
helm templateor a GitOps tool such as Argo CD, see Compile worker certificates.Take the steps in Secure the compile workers.
Check before you upgrade
The nodes, the namespace’s resource quota, and any LimitRange must have room for the two compilers in the CMF pod: with the default
resources, 4 GiB more memory, 2 more CPU of requests, and 4 more CPU of limits. Kubernetes stops the old CMF pod first, so without room CMF is down until there is room. With artifact management and the environment catalog on, the compile worker pod also needs room, orhelm upgrade --waittimes out. By default it takes 3 GiB of memory and the CPU request and limit of thecmfcontainer, 1 and 2 CPU with the defaultresources. See Shared compilers and Size the compile workers.If NetworkPolicies in the CMF namespace deny traffic by default, allow traffic from the CMF pod to the compile worker pod on ports
8090and up, and from the worker pod to CMF’s internal port8081, Kafka, and Schema Registry. Otherwise every statement of an environment with artifacts waits forcompileTimeoutand fails. See Limit network access.cmf.metrics.port,cmf.mds.port, and the containers inextraContainersmust not use ports8081,8095, or8096.If
deploymentOverridesreplaces the pod’svolumesor a container’svolumeMounts, add the compiler entries that the chart creates to your lists. Runhelm templatewith your values to see them.If
extraEnvsets a JVM option variable such asJAVA_TOOL_OPTIONSwithvalueFrom, check it as described in What compilers take from the cmf container.Every compute pool on Flink 1.19 must run a
cp-flink-sqlimage of1.19-cp8or later. Change theimageof older compute pools before you create statements on them; see Update a compute pool.If artifact management and the environment catalog are both on but no SQL statement uses an artifact, the statements of environments that have any artifact, such as an application JAR, still compile on the compile worker that the chart adds. Either set up the worker as described above, or set
cmf.statementService.workers.count: 0.
After you upgrade
Statements that were running before the upgrade keep running. CMF compiles a statement only when it is created.
CMF can be ready before its compilers have started. A statement created in that time fails with an error that the compiler cannot be reached; create it again.
The
cmf.statement.compilation.*metrics count only statements that thecmfcontainer compiles itself, so dashboards and alerts that use them no longer see the statements that the shared compilers and compile workers compile.The
cmf.artifacts.cache.*settings are no longer used. CMF ignores them and logs a warning.
Statements with a startFromSavepoint
Earlier versions did not record when a statement’s job started from its startFromSavepoint.
When you upgrade, CMF assumes that each statement with a startFromSavepoint started from it
at its last update. When it recreates the statement’s Flink resource, as described in
How CMF chooses the restore point, it therefore counts only savepoints taken after that update as the
statement’s latest savepoint. If a newer savepoint was taken before the last update, CMF
restores the startFromSavepoint instead, and logs the warning Restoring from the applied
startFromSavepoint although a newer savepoint exists, which names the newer savepoint. To
restore that savepoint after recovery, update the statement with startFromSavepoint set to it
and a new savepointRedeployNonce, which redeploys the statement from it.
To avoid this, take a savepoint of each such statement after you upgrade to CMF 2.5.0. Recovery then restores that savepoint, or a later one.