Upgrade Confluent Manager for Apache Flink

Flink applications are deployed in Confluent Platform with Confluent Manager for Apache Flink®, which is a central management component that enables users to securely manage a fleet of Flink applications across multiple environments. Use these instructions to upgrade earlier versions of Confluent Manager for Apache Flink (CMF) to a later version. To install CMF, see Install Confluent Manager for Apache Flink with Helm.

Warning

Downgrades are not supported by CMF. Before upgrading, ensure you have tested the new version in a non-production environment.

Important

If you upgrade to CMF 2.5 from an earlier version, first complete the checks in Upgrade to CMF 2.5.

If you ran CMF 2.3 or 2.4 with shared compute pools, read Delete duplicate statement clusters left by CMF 2.3 and 2.4 before you start: its first step must be done while CMF still runs.

Follow these steps to upgrade CMF:

  1. Scale down CMF to zero (0) pods and wait for the pod to be deleted. The following command shows an example of how to scale down CMF:

    kubectl scale deployment confluent-manager-for-apache-flink --replicas=0 -n <namespace>
    kubectl wait --for=delete pod -l app.kubernetes.io/name=confluent-manager-for-apache-flink --timeout=60s -n <namespace>
    
  2. Uninstall the existing Flink Kubernetes Operator:

    helm uninstall cp-flink-kubernetes-operator -n <namespace>
    
  3. Ensure certificate manager is installed, or use the following command to install it:

    kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.18.2/cert-manager.yaml
    kubectl wait --for=condition=Available deployment -n cert-manager cert-manager-webhook --timeout=180s
    
  4. Find the version Confluent Platform for Apache Flink Kubernetes operator you want to install. For a list of available versions, see the Versions and Interoperability for Confluent Manager for Apache Flink.

    helm repo add confluentinc https://packages.confluent.io/helm
    helm repo update confluentinc
    helm upgrade --install --version "<desired_version>" cp-flink-kubernetes-operator confluentinc/flink-kubernetes-operator
    
  5. Upgrade CMF to the desired version using the Helm command:

    helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink --namespace <namespace> ...
    

Important

If the CMF database is file system based, using helm uninstall cmf will lead to the loss of the database because the Persistent Volume Claim (PVC) is managed by CMF.

Delete duplicate statement clusters left by CMF 2.3 and 2.4

Each time they started, CMF 2.3.0 to 2.3.2 and 2.4.0 to 2.4.2 created or replaced a standalone FlinkDeployment with the name of each running, pending, failing, or stopped statement on a shared compute pool. Unless the statement was stopped the last time one of these versions started, this cluster can run a second copy of the statement’s job, which reads the same sources and writes to the same sinks. Stopping, resuming, or deleting the statement does not stop it. CMF can also take the status and errors of such a cluster for those of the statement, and attach to the statement the savepoints that the Flink Kubernetes Operator takes of the cluster. Recovery in CMF 2.5.0 can then restore those savepoints.

Later 2.3 and 2.4 versions create such a cluster only when a read of a compute pool from the database fails once while CMF starts. They then log Unable to determine compute pool type for statement. CMF 2.5.0 never creates them, and no version cleans them up. If you ran CMF 2.3 or 2.4 with shared compute pools, delete them as soon as possible. While you still run 2.3.0 to 2.3.2 or 2.4.0 to 2.4.2, which create them again each time they start, delete them as part of the upgrade to a version that does not. Those are 2.3.3 and later 2.3 versions, 2.4.3 and later 2.4 versions, and 2.5.0 and later. On such a version, you can also delete them on their own:

  1. Right before you scale CMF down, while it still runs, note the namespace and the Kubernetes cluster of each environment (kubernetesNamespace and kubernetesClusterName), and the names of the environment’s statements on dedicated compute pools, in any phase. You need this information in step 4, while CMF is down. A statement’s compute pool is in its spec.computePoolName, and the spec.type of a compute pool is DEDICATED or SHARED. Do not create statements until you have finished step 4.

  2. Scale CMF down to zero pods, as in step 1 of the upgrade: while it runs, CMF can take the status of a cluster that you delete for that of the statement. Delete the clusters before you uninstall the Flink Kubernetes Operator: without the operator, the deletion of a FlinkDeployment does not complete, and its job keeps running.

  3. On the Kubernetes cluster of each environment, list the FlinkDeployments that CMF created for statements in the environment’s namespace, with the environment of each in the ENVID column. Include the namespaces of environments that you deleted: deleting an environment does not delete these clusters.

    kubectl get flinkdeployment -n <namespace> -l confluent.io/cmfResource=Statement -L confluent.io/envId
    
  4. Delete each listed FlinkDeployment whose ENVID is the name of one of your environments, or of one that you deleted, and whose name is not that of a statement on a dedicated compute pool in that environment:

    kubectl delete flinkdeployment <name> -n <namespace>
    

    Note the following:

    • A statement on a dedicated compute pool runs as the FlinkDeployment with its name, so do not delete those.

    • A statement whose compute pool no longer exists is FAILED or COMPLETED and cannot run again, so you can delete its FlinkDeployment too.

    • If an application has the name of a statement on a shared compute pool, the listed FlinkDeployment of that name replaced the application’s own. When CMF starts again, it recreates the application, from the savepoint that its spec names, or without state.

    • Leave the other listed FlinkDeployments, which belong to other CMF installations that use the namespace. If such an installation has an environment with the name of one of yours, these labels do not tell its FlinkDeployments from yours. Check with its administrators before you delete any.

    • The FlinkDeployments of applications and the session clusters of compute pools have another value of confluent.io/cmfResource, and are not listed.

  5. Continue with step 2 of the upgrade, or, if you delete the clusters on their own, scale CMF back up.

  6. After CMF runs again, delete the savepoints that CMF attached from the deleted clusters. For each statement on a shared compute pool, list the savepoints that were taken of a FlinkDeployment. The response is paginated: use the page parameter for more savepoints.

    curl -s "http://cmf:8080/cmf/api/v1/environments/<environment>/statements/<statement>/savepoints?size=100" \
      | jq -r '.items[] | select(.metadata.labels["job-reference.kind"] == "FlinkDeployment") | .metadata.name'
    

    Delete each listed savepoint with force=true:

    curl -X DELETE "http://cmf:8080/cmf/api/v1/environments/<environment>/statements/<statement>/savepoints/<savepoint>?force=true"
    

    Then delete the FlinkStateSnapshots that remain from each FlinkDeployment that you deleted in step 4, so that CMF does not attach them to a statement or application of the same name later:

    kubectl delete flinkstatesnapshot -n <namespace> -l job-reference.kind=FlinkDeployment,job-reference.name=<name>
    

Upgrade to CMF 2.5

Starting with version 2.5, CMF compiles SQL statements in shared compilers, which run as two extra containers in the CMF pod, and statements that use artifacts compile only on compile workers. For details, see Configure SQL Compilers and Compile Workers for Confluent Manager for Apache Flink.

CMF 2.5.0 can recover Flink SQL statements after the loss of their Kubernetes cluster from copies of their compiled plans, which it stores in artifact storage. To protect the statements that exist when you upgrade, enable artifact management, and wait until CMF has copied their plans while their cluster still exists. Then take a savepoint of each statement that has a startFromSavepoint, as described in Statements with a startFromSavepoint. For all the steps, see Prepare for Disaster Recovery in Confluent Manager for Apache Flink.

If you return to an earlier version by restoring a backup of the CMF database that you took before the upgrade, that version neither uses nor deletes the plans that CMF 2.5.0 stored under statement-plans/ in artifact storage, and you can delete them.

Set up compile workers

If your SQL statements use artifacts, such as user-defined functions or custom connectors, with cmf.artifacts.enabled and cmf.sql.environmentCatalog.enabled both true, then after the upgrade every statement of an environment that has an artifact compiles on a compile worker, including statements that use no artifact. The chart adds one compile worker by default. Before you upgrade, set up the compile workers:

  • Copy the environment variables and volumes from the top-level extraEnv and mountedVolumes that your catalogs and connectors use into cmf.statementService.workers.extraEnv and cmf.statementService.workers.mountedVolumes. Mark each volume mount readOnly: true, and leave out JVM option variables such as JAVA_TOOL_OPTIONS. Otherwise the chart refuses the values. See Give the workers what catalogs and connectors need.

  • Copy the JVM options that catalogs need, such as a trust store, from the top-level jvmArgs to cmf.statementService.workers.compilerJvmArgs, and keep a heap size there, such as -Xmx1g.

  • Workers do not get the cloud identity of the CMF ServiceAccount. A catalog that uses it needs credentials in its options, or a ServiceAccount for the workers.

  • If you set the top-level nodeSelector, tolerations, affinity, topologySpreadConstraints, priorityClassName, podLabels, or podAnnotations, copy what the worker pod needs into workerDeploymentOverrides; see Compile worker settings. The worker pod gets none of them. If your nodes are all tainted, or admission policies require pod labels, the worker pod stays Pending, helm upgrade --wait times out, and --atomic rolls back the upgrade. If no SQL statement uses an artifact, you can instead set cmf.statementService.workers.count: 0.

  • Set cmf.statementService.workers.count for the load; see Size the compile workers.

  • If you render the chart with helm template or a GitOps tool such as Argo CD, see Compile worker certificates.

  • Take the steps in Secure the compile workers.

Check before you upgrade

  • The nodes, the namespace’s resource quota, and any LimitRange must have room for the two compilers in the CMF pod: with the default resources, 4 GiB more memory, 2 more CPU of requests, and 4 more CPU of limits. Kubernetes stops the old CMF pod first, so without room CMF is down until there is room. With artifact management and the environment catalog on, the compile worker pod also needs room, or helm upgrade --wait times out. By default it takes 3 GiB of memory and the CPU request and limit of the cmf container, 1 and 2 CPU with the default resources. See Shared compilers and Size the compile workers.

  • If NetworkPolicies in the CMF namespace deny traffic by default, allow traffic from the CMF pod to the compile worker pod on ports 8090 and up, and from the worker pod to CMF’s internal port 8081, Kafka, and Schema Registry. Otherwise every statement of an environment with artifacts waits for compileTimeout and fails. See Limit network access.

  • cmf.metrics.port, cmf.mds.port, and the containers in extraContainers must not use ports 8081, 8095, or 8096.

  • If deploymentOverrides replaces the pod’s volumes or a container’s volumeMounts, add the compiler entries that the chart creates to your lists. Run helm template with your values to see them.

  • If extraEnv sets a JVM option variable such as JAVA_TOOL_OPTIONS with valueFrom, check it as described in What compilers take from the cmf container.

  • Every compute pool on Flink 1.19 must run a cp-flink-sql image of 1.19-cp8 or later. Change the image of older compute pools before you create statements on them; see Update a compute pool.

  • If artifact management and the environment catalog are both on but no SQL statement uses an artifact, the statements of environments that have any artifact, such as an application JAR, still compile on the compile worker that the chart adds. Either set up the worker as described above, or set cmf.statementService.workers.count: 0.

After you upgrade

  • Statements that were running before the upgrade keep running. CMF compiles a statement only when it is created.

  • CMF can be ready before its compilers have started. A statement created in that time fails with an error that the compiler cannot be reached; create it again.

  • The cmf.statement.compilation.* metrics count only statements that the cmf container compiles itself, so dashboards and alerts that use them no longer see the statements that the shared compilers and compile workers compile.

  • The cmf.artifacts.cache.* settings are no longer used. CMF ignores them and logs a warning.

Statements with a startFromSavepoint

Earlier versions did not record when a statement’s job started from its startFromSavepoint. When you upgrade, CMF assumes that each statement with a startFromSavepoint started from it at its last update. When it recreates the statement’s Flink resource, as described in How CMF chooses the restore point, it therefore counts only savepoints taken after that update as the statement’s latest savepoint. If a newer savepoint was taken before the last update, CMF restores the startFromSavepoint instead, and logs the warning Restoring from the applied startFromSavepoint although a newer savepoint exists, which names the newer savepoint. To restore that savepoint after recovery, update the statement with startFromSavepoint set to it and a new savepointRedeployNonce, which redeploys the statement from it.

To avoid this, take a savepoint of each such statement after you upgrade to CMF 2.5.0. Recovery then restores that savepoint, or a later one.