# Upgrade Confluent for Kubernetes Upgrade Confluent for Kubernetes (CFK) by verifying your environment, setting the required compatibility annotations, and running the Helm upgrade. After you upgrade CFK, upgrade your Confluent Platform components as described in [Upgrade Confluent Platform Using Confluent for Kubernetes](co-upgrade-cp.md#co-upgrade-cp). Before you start the upgrade, review the [pre-upgrade tasks and known issues](co-upgrade-considerations.md#co-upgrade-additional-considerations) so the upgrade is not interrupted. For configurations that cannot be updated on deployed clusters, see [Configurations that can’t change after deployment](co-plan-availability.md#co-update). ## Prerequisites Confirm the following before you begin: - `kubectl` is installed and pointed at the cluster where CFK is deployed: ```bash kubectl config current-context ``` Expected: The context name of your target cluster. - Helm 3.x or 4.x is installed: ```bash helm version ``` Expected: `version.BuildInfo{Version:"v3.x.x", ...}` or `version.BuildInfo{Version:"v4.x.x", ...}`. - You know the namespace where CFK is deployed. Replace `` in the commands on this page with that value. - Know your current CFK version so you can confirm the upgrade afterward: ```bash helm list -n ``` Expected: the `confluent-operator` row shows your current `CHART` and `APP VERSION`. ## Upgrade CFK Follow the steps in this section to upgrade CFK using Helm: ![Flowchart of the eight |co| upgrade steps, showing the operator upgrade and init container update staged while reconciliation is blocked, applied together in step 8](images/co-upgrade-cfk-steps-flow.svg) ### Step 1: Set compatibility annotations (Confluent Platform 7.x only) If you are running Confluent Platform 8.x, skip this step. You need these annotations only to run Confluent Platform 7.x with CFK 3.x. If you are upgrading from CFK 2.x to 3.x with Confluent Platform 7.x, set the annotations as shown below. If you are upgrading between CFK 3.x versions with Confluent Platform 7.x, you should have already set the annotations during your earlier upgrade. Verify they are still present, and re-apply them if missing. - Set the Log4j annotation on each deployed Confluent Platform component (the component list is in [Step 2](#co-upgrade-disable-reconcile)). You need the `platform.confluent.io/use-log4j1=true` annotation to use Confluent Platform 7.x with CFK 3.0 or later. Replace `` with the custom resource (CR) kind (for example, `kafka`) and `` with the CR name. ```bash kubectl annotate \ platform.confluent.io/use-log4j1=true \ -n ``` Expected: `.platform.confluent.io/ annotated` Verify the annotation is set on each component: ```bash kubectl get -n \ -o jsonpath='{.metadata.annotations.platform\.confluent\.io/use-log4j1}' ``` Expected: `true` - *Only if your deployment uses basic authentication*, set the Java Authentication and Authorization Service (JAAS) class path annotation for Confluent Platform 7.x compatibility. Apply only to the web components: Control Center, Control Center (Legacy), Schema Registry, Connect, ksqlDB, and REST Proxy. If your deployment does not use basic authentication, skip this annotation. ```bash kubectl annotate \ platform.confluent.io/use-old-jetty9=true \ -n ``` Expected: `.platform.confluent.io/ annotated` #### NOTE Without this annotation, you cannot log in to Confluent Platform 7.x components using basic authentication, and you get a login prompt loop in Control Center. For more information, see [Issue: Authentication failures when using CFK 3.0 with Confluent Platform 7.x](co-troubleshoot-upgrade.md#co-jaas-class-change). ### Step 2: Disable resource reconciliation Disable reconciliation for every Day-1 and Day-2 custom resource (CR) kind that you have deployed in each namespace where you have deployed Confluent Platform. This prevents rolling restarts during the CFK upgrade. Run the following command for each CR, replacing `` and `` with the resource type and name: ```bash kubectl annotate \ platform.confluent.io/block-reconcile=true \ -n ``` For example: ```bash kubectl annotate kafka kafka \ platform.confluent.io/block-reconcile=true \ -n ``` Expected: `.platform.confluent.io/ annotated` Day-1 CRs include, for example, `Connect`, `ControlCenter`, `Kafka`, `KafkaRestProxy`, `KsqlDB`, `SchemaRegistry`, and either `KRaftController` for KRaft-based Confluent Platform or `Zookeeper` for ZooKeeper-based Confluent Platform. Apply the annotation to every Day-1 CR you have deployed, including any kind added in a later CFK version. Repeat for all deployed Day-1 CRs: | Component | CR kind | |-------------------------------------|-------------------| | Apache Kafka® | `kafka` | | Kafka Connect | `connect` | | Control Center (Legacy) | `controlcenter` | | Confluent REST Proxy | `kafkarestproxy` | | ksqlDB | `ksqldb` | | Schema Registry | `schemaregistry` | | KRaft Controller (KRaft-based only) | `kraftcontroller` | | ZooKeeper (ZooKeeper-based only) | `zookeeper` | Day-2 CRs include, for example, `KafkaTopic`, `Schema`, `SchemaExporter`, `Connector`, `ClusterLink`, and `ConfluentRolebinding`. Apply the same annotation to every Day-2 CR you have deployed. Verify reconciliation is disabled on each CR: ```bash kubectl get -n \ -o jsonpath='{.metadata.annotations.platform\.confluent\.io/block-reconcile}' ``` Expected: `true` ### Step 3: Update the CFK Helm repository To update the CFK Helm repository, run the following commands: ```bash helm repo add confluentinc https://packages.confluent.io/helm ``` ```bash helm repo update ``` Expected: `Update Complete. ⎈Happy Helming!⎈` ### Step 4: Pull the CFK chart Pull the chart using one of the following options. ### Latest version ```bash helm pull confluentinc/confluent-for-kubernetes --untar ``` ### Specific version Get the image tag from [Confluent for Kubernetes image tags](co-supported-environments.md#co-operator-image-tags): ```bash helm pull confluentinc/confluent-for-kubernetes --version --untar ``` ### Download bundle For details, see [Deploy CFK using the download bundle](co-deploy-operator.md#co-download-bundle). The `--untar` flag unpacks the chart into a `confluent-for-kubernetes/` directory in your current location. Run Step 5 (and the `helm upgrade`) from this same directory so the `confluent-for-kubernetes/crds/` path resolves. Expected: a `confluent-for-kubernetes/` directory containing `crds/`, `templates/`, `Chart.yaml`, and `values.yaml`. ### Step 5: Upgrade Confluent Platform Custom Resource Definitions (CRDs) Apply the CRDs from the chart directory. Helm does not upgrade CRDs automatically, so you must do this step manually. For details, see the [Helm documentation](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations). CFK provides the CRDs in two locations: * `crds/`, at the root of the [download bundle](co-deploy-operator.md#co-download-bundle). These CRDs include full field descriptions (about 4.7 MB). Use these CRDs, because the descriptions make `kubectl explain` and CR validation more useful. * `confluent-for-kubernetes/crds/`, inside the CFK Helm chart. This location is available whether you got the chart from the Helm repository or from the download bundle, but the path differs by source. If you got the chart from the Helm repository, the path is `confluent-for-kubernetes/crds/`. If you got the chart from the download bundle, the chart is nested one level deeper, so the path is `helm/confluent-for-kubernetes/crds/`. These CRDs have field descriptions removed (about 1.8 MB) so that Helm can track them within its `ConfigMap` size limit. Use server-side apply. It upgrades the CRDs cleanly without the client-side annotation-size limit or the per-CRD `last-applied-configuration` warnings. If you have the download bundle, apply the full CRDs from the root-level `crds/` directory: ```bash kubectl apply --server-side=true -f crds/ ``` If you only have the Helm chart, apply the description-stripped CRDs from the chart directory instead: ```bash kubectl apply --server-side=true -f confluent-for-kubernetes/crds/ ``` Expected: Each CRD shows `serverside-applied`. If you run the `kubectl apply` command without the `--server-side=true` flag and get an error similar to the following: ```text The CustomResourceDefinition "kafkas.platform.confluent.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes make: *** [install-crds] Error 1 ``` Re-run the command with the `--server-side=true` flag. If the server-side apply returns a conflict error: ```text Apply failed with 1 conflict: conflict with "helm" using apiextensions.k8s.io/v1: .spec.versions ``` Re-run with `--force-conflicts`: ```bash kubectl apply --server-side=true --force-conflicts -f confluent-for-kubernetes/crds/ ``` #### NOTE You can also use client-side apply (`kubectl apply -f confluent-for-kubernetes/crds/`), but it prints a missing `last-applied-configuration` warning for each CRD (patched automatically) and can fail on large CRDs with `metadata.annotations: Too long: must have at most 262144 bytes`. Server-side apply avoids both. ### Step 6: Upgrade CFK Upgrade the operator using the option that matches your deployment. ### Custom values file Run the upgrade with `--dry-run` first to preview the changes. For details about the values file, see [Deploy CFK with Custom Values](co-deploy-custom-values.md#co-values-file). ```bash helm upgrade --install confluent-operator \ confluentinc/confluent-for-kubernetes \ --values \ -n \ --dry-run ``` After you review the output, apply the upgrade. ```bash helm upgrade --install confluent-operator \ confluentinc/confluent-for-kubernetes \ --values \ -n ``` ### No values file ```bash helm upgrade --install confluent-operator \ confluentinc/confluent-for-kubernetes \ -n ``` ### Global license If you use a global license (`globalLicense: true`), specify the license key on the upgrade command. For details, see [Update CFK global license](co-license.md#co-licence-global-level). ```bash helm upgrade --install confluent-operator \ confluentinc/confluent-for-kubernetes \ --values values.yaml \ -n \ --set licenseKey= ``` ### Download bundle Follow the steps in [Deploy CFK using the download bundle](co-deploy-operator.md#co-download-bundle). Expected: `Release "confluent-operator" has been upgraded. Happy Helming!` After the upgrade completes, verify that the operator pod is healthy before you continue. Resource reconciliation remains blocked, so your Confluent Platform components are not affected yet. To upgrade to a specific version (hotfix or patch), choose one of the following options: - If you use a custom values file, update the `image.tag` in `values.yaml` and then run the `helm upgrade` command above with `--values `. ```yaml image: tag: "" ``` - If you do not use a custom values file, pass the target version with `--version` on the upgrade command. ```bash helm upgrade --install confluent-operator \ confluentinc/confluent-for-kubernetes \ --version \ -n ``` ### Step 7: Upgrade CFK init container Stage this change while resource reconciliation is still blocked. When you enable reconciliation in the next step, a single rolling restart applies both the operator upgrade and the new init container image. In each Confluent Platform component CR, update the CFK init container image tag to the tag for your target CFK version. The example below uses `3.3.0`, which is the canonical tag for the current CFK minor version. For the exact tag for any CFK release (including patch releases), see the [Confluent for Kubernetes image tags](co-supported-environments.md#co-operator-image-tags) reference table. ```yaml kind: spec: image: init: confluentinc/confluent-init-container:3.3.0 ``` #### IMPORTANT Do not skip this step. Starting in CFK 3.3.0, the init container delivers the `cfkprober` binary that the new `exec` liveness probe runs. If you do not update the init container image tag, the older init container does not deliver the `cfkprober` binary. As a result, the liveness probe fails open and reports the component as live even when it is not. For details, see [Liveness probe change to exec in CFK 3.3.0](co-upgrade-overview.md#co-upgrade-liveness-exec-probe). Apply the change to each component CR with `kubectl patch` (or edit the CR directly with `kubectl edit`). For example, for Kafka: ```bash kubectl patch kafka -n --type merge \ -p '{"spec":{"image":{"init":"confluentinc/confluent-init-container:3.3.0"}}}' ``` Expected: `kafka.platform.confluent.io/ patched` ### Step 8: Enable resource reconciliation Re-enable reconciliation for every Day-1 and Day-2 CR that you disabled in Step 2. For each CR, run one of the following commands. The first command sets the annotation to `false` so CFK can clear the `ReconcileIsBlocked` status condition. The second command removes the annotation entirely: ```bash kubectl annotate \ platform.confluent.io/block-reconcile=false \ -n --overwrite ``` **OR** ```bash kubectl annotate \ platform.confluent.io/block-reconcile- \ -n ``` Verify the annotation no longer blocks reconciliation: ```bash kubectl get -n \ -o jsonpath='{.metadata.annotations.platform\.confluent\.io/block-reconcile}' ``` Expected: empty output if you removed the annotation, or `false` if you set it to `false`. #### NOTE When reconciliation resumes, the upgraded CFK reconciles each component, applying both the operator upgrade and the staged init container image update in a single rolling restart. You might briefly see pods restart or show `Error` or `CrashLoopBackOff` while they reconnect to their dependencies. For example, Schema Registry may crash while Kafka is restarting. The pods recover automatically. Wait for each component CR to return to `STATUS: RUNNING` before treating the upgrade as failed. ### Final verification After completing all steps, verify the CFK Operator and all Confluent Platform component CRs are healthy on the new version. - Verify the CFK chart and pod version: ```bash helm list -n ``` Expected: `confluent-operator` shows the updated chart version. ```bash kubectl get pods -n | grep confluent-operator ``` Expected: The `confluent-operator` pod is in `Running` state. - Verify each Confluent Platform component CR has returned to `RUNNING`: ```bash kubectl get -n ``` Expected: `STATUS` column shows `RUNNING`. - Verify the init container tag actually in use matches your target: ```bash kubectl get pod -n \ -o jsonpath='{.spec.initContainers[*].image}' ``` Expected: the image tag matches what you set in Step 7 (for example, `confluentinc/confluent-init-container:3.3.0`). ## Next steps Now that CFK is upgraded, upgrade your Confluent Platform components. For instructions, see [Upgrade Confluent Platform Using Confluent for Kubernetes](co-upgrade-cp.md#co-upgrade-cp).