Upgrade Confluent for Kubernetes

Before you start the upgrade process, make sure your Kubernetes cluster is among the Supported Environments for the target version of Confluent for Kubernetes (CFK).

Warning

Do not upgrade Confluent Platform server image and configuration updates simultaneously, as this can stop your upgrades.

For best practices, see Avoid simultaneous changes during upgrades.

Upgrade CFK

Review Upgrade considerations and troubleshooting and address any required steps.
If you are upgrading your CFK 2.x to 3.x to deploy and manage Confluent Platform 7.x:
- For Log4J, set the annotation for the components you want to use Log4J:
```
kubectl annotate <CR kind> <CR name> \
    platform.confluent.io/use-log4j1=true \
     --namespace <namespace>
```
  The platform.confluent.io/use-log4j1=true annotation is required to use Confluent Platform 7.x with CFK 3.0+.
- To use the JAAS class path compatible with Confluent Platform 7.x in basic authentication, set the annotation, platform.confluent.io/use-old-jetty9=true, to your Confluent Platform component CRs, such as Control Center, Control Center (Legacy), Schema Registry, Connect, ksqlDB, and REST Proxy:
```
kubectl annotate <CR kind> <CR name> \
    platform.confluent.io/use-old-jetty9=true \
     --namespace <namespace>
```
  If you do not set the annotation properly, you will not be able to log into the Confluent Platform 7.x components using basic authentication, and you will get a login prompt loop when try to login to Control Center.
  For details, see Issue: Authentication failures when using CFK 3.0 with Confluent Platform 7.x.

Disable resource reconciliation.

To prevent Confluent Platform components from rolling restarts, temporarily disable resource reconciliation of the components in each namespace where you have deployed Confluent Platform, specifying the CR kinds and CR names:

kubectl annotate connect connect \
    platform.confluent.io/block-reconcile=true \
     --namespace <namespace>

kubectl annotate controlcenter controlcenter \
     platform.confluent.io/block-reconcile=true \
     --namespace <namespace>

kubectl annotate kafkarestproxy kafkarestproxy \
     platform.confluent.io/block-reconcile=true \
     --namespace <namespace>

kubectl annotate kafka kafka \
     platform.confluent.io/block-reconcile=true \
     --namespace <namespace>

kubectl annotate ksqldb ksqldb \
     platform.confluent.io/block-reconcile=true \
     --namespace <namespace>

kubectl annotate schemaregistry schemaregistry \
     platform.confluent.io/block-reconcile=true \
     --namespace <namespace>

For KRaft-based Confluent Platform:

kubectl annotate kraftcontroller kraftcontroller \
     platform.confluent.io/block-reconcile=true \
     --namespace <namespace>

For ZooKeeper-based Confluent Platform:

kubectl annotate zookeeper zookeeper \
     platform.confluent.io/block-reconcile=true \
     --namespace <namespace>

Add the CFK Helm repo:

helm repo add confluentinc https://packages.confluent.io/helm

helm repo update

Get the CFK chart.
- From the Helm repo:
  - To get the latest CFK chart:
```
helm pull confluentinc/confluent-for-kubernetes --untar
```
  - To get a specific version of the CFK chart, get the image tag of the CFK version from Confluent for Kubernetes image tags, and specify the version tag with the --version flag:
```
helm pull confluentinc/confluent-for-kubernetes --version <CFK image tag> --untar
```
- From a download bundle as specified in Deploy CFK using the download bundle.
IMPORTANT. Upgrade Confluent Platform custom resource definitions (CRDs).
This step is required because Helm does not support upgrading or deleting CRDs using Helm. For more information, see the Helm documentation.
CFK provides the CRDs in two locations:
- crds/, at the root of the download bundle. These CRDs include full field descriptions (about 4.7 MB) and are recommended, because the descriptions make kubectl explain and CR validation more useful.
- confluent-for-kubernetes/crds/, inside the CFK Helm chart. This location is available whether you got the chart from the Helm repository or from the download bundle, but the path differs by source. If you got the chart from the Helm repository, the path is confluent-for-kubernetes/crds/. If you got the chart from the download bundle, the chart is nested one level deeper, so the path is helm/confluent-for-kubernetes/crds/. These CRDs have field descriptions removed (about 1.8 MB) so that Helm can track them within its ConfigMap size limit.
If you have the download bundle, apply the full CRDs from the root-level crds/ directory using server-side apply, because their size can otherwise exceed the kubectl client-side annotation limit:
```
kubectl apply --server-side=true -f crds/
```
If you only have the Helm chart, apply the description-stripped CRDs instead, using server-side apply here as well. Use whichever path from above matches how you got the chart:
```
kubectl apply --server-side=true -f confluent-for-kubernetes/crds/
```
1. If you ran the kubectl apply command on the description-stripped CRDs without the --server-side=true flag and got an error similar to the below:
```
The CustomResourceDefinition "kafkas.platform.confluent.io" is invalid:
metadata.annotations: Too long: must have at most 262144 bytes make: ***
[install-crds] Error 1
```
  Run the following commands:
```
kubectl apply --server-side=true -f <CRD>
```
2. If running kubectl apply with the --server-side=true flag returns an error similar to the below:
```
Apply failed with 1 conflict: conflict with "helm" using
apiextensions.k8s.io/v1: .spec.versions Please review the fields
above--they currently have other managers.
```
  Run kubectl apply with an additional flag, --force-conflicts:
```
kubectl apply --server-side=true --force-conflicts -f <CRD>
```
Upgrade CFK to 3.3.0.
- If you deployed customized CFK using the values file:
  For debugging and validating upgrades, when you have a custom values file, it is recommended that you run the helm upgrade command with the --dry-run flag to preview. The command with the --dry-run flag will not actually apply the changes to the cluster, but will print the Kubernetes manifests that would be applied.
```
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values <path-to-values-file> \
  --namespace <namespace> \
  --dry-run
```
  After reviewing the Kubernetes manifests, run the following command to upgrade CFK:
```
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values <path-to-values-file> \
  --namespace <namespace>
```
- If you deployed CFK without customizing the values file, run the following command to upgrade CFK:
```
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --namespace <namespace>
```
- If you deployed CFK from a download bundle, upgrade CFK as specified in Deploy CFK using the download bundle.
Note that when using the CFK global license (globalLicense: true in the component CRs), you need to specify the license key in the helm upgrade command using the --set licenseKey=<CFK license key> flag. For details, see Update CFK global license.
```
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values values.yaml \
  --set licenseKey=<CFK license key>
```
After the upgrade completes, verify that the operator pod is healthy before you continue. Resource reconciliation remains blocked, so your Confluent Platform components are not affected yet.
Alternatively, upgrade CFK to a specific version, such as a hotfix or a patch version.
- If you deployed CFK using the values file, in your values.yaml, update the CFK image.tag to the image tag of the CFK version specified in Confluent for Kubernetes image tags:
```
image:
  tag: "<CFK image tag>"
```
  And run the following command to upgrade CFK:
```
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values <path-to-values-file> \
  --namespace <namespace>
```
- If you did not use a customized values.yaml for CFK deployment, run the following command to upgrade CFK to a specific version, using the image tag of the CFK version specified in Confluent for Kubernetes image tags:
```
helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --version <CFK image tag>
  --namespace <namespace>
```
Upgrade the CFK init container.
Stage this change while resource reconciliation is still blocked. When you enable reconciliation in the next step, a single rolling restart applies both the operator upgrade and the new init container image.

Enable resource reconciliation for each Confluent Platform component for which you turned off reconciliation in the earlier step:

kubectl annotate <component CR kind> <cluster name> \
  platform.confluent.io/block-reconcile- \
  --namespace <namespace>

Upgrade CFK init container

In each Confluent Platform component CR, update the CFK init container image tag to the version of CFK you are upgrading to, 3.3.0:

kind: <Confluent component>
spec:
  image:
    init: confluentinc/confluent-init-container:3.3.0

Important

Do not skip this step. Starting in CFK 3.3.0, the init container delivers the cfkprober binary that the new exec liveness probe runs. If you do not update the init container image tag, the older init container does not deliver the cfkprober binary, and the liveness probe fails open and reports the component as live even when it is not. For details, see Liveness probe change to exec in CFK 3.3.0.