Upgrade Confluent for Kubernetes

Upgrade Confluent for Kubernetes (CFK) by verifying your environment, setting the required compatibility annotations, and running the Helm upgrade. After you upgrade CFK, upgrade your Confluent Platform components as described in Upgrade Confluent Platform Using Confluent for Kubernetes.

Before you start the upgrade, review the pre-upgrade tasks and known issues so the upgrade is not interrupted. For configurations that cannot be updated on deployed clusters, see Configurations that can’t change after deployment.

Prerequisites

Confirm the following before you begin:

kubectl is installed and pointed at the cluster where CFK is deployed:
```
kubectl config current-context
```
Expected: The context name of your target cluster.
Helm 3.x or 4.x is installed:
```
helm version
```
Expected: version.BuildInfo{Version:"v3.x.x", ...} or version.BuildInfo{Version:"v4.x.x", ...}.
You know the namespace where CFK is deployed. Replace <namespace> in the commands on this page with that value.
Know your current CFK version so you can confirm the upgrade afterward:
```
helm list -n <namespace>
```
Expected: the confluent-operator row shows your current CHART and APP VERSION.

Upgrade CFK

Follow the steps in this section to upgrade CFK using Helm:

Flowchart of the eight |co| upgrade steps, showing the operator upgrade and init container update staged while reconciliation is blocked, applied together in step 8 — Flowchart of the eight CFK upgrade steps, showing the operator upgrade and init container update staged while reconciliation is blocked, applied together in step 8

Step 1: Set compatibility annotations (Confluent Platform 7.x only)

If you are running Confluent Platform 8.x, skip this step. You need these annotations only to run Confluent Platform 7.x with CFK 3.x.

If you are upgrading from CFK 2.x to 3.x with Confluent Platform 7.x, set the annotations as shown below. If you are upgrading between CFK 3.x versions with Confluent Platform 7.x, you should have already set the annotations during your earlier upgrade. Verify they are still present, and re-apply them if missing.

Set the Log4j annotation on each deployed Confluent Platform component (the component list is in Step 2). You need the platform.confluent.io/use-log4j1=true annotation to use Confluent Platform 7.x with CFK 3.0 or later. Replace <CR kind> with the custom resource (CR) kind (for example, kafka) and <CR name> with the CR name.
```
kubectl annotate <CR kind> <CR name> \
    platform.confluent.io/use-log4j1=true \
    -n <namespace>
```
Expected: <CR kind>.platform.confluent.io/<CR name> annotated

Verify the annotation is set on each component:
```
kubectl get <CR kind> <CR name> -n <namespace> \
    -o jsonpath='{.metadata.annotations.platform\.confluent\.io/use-log4j1}'
```
Expected: true
Only if your deployment uses basic authentication, set the Java Authentication and Authorization Service (JAAS) class path annotation for Confluent Platform 7.x compatibility. Apply only to the web components: Control Center, Control Center (Legacy), Schema Registry, Connect, ksqlDB, and REST Proxy. If your deployment does not use basic authentication, skip this annotation.
```
kubectl annotate <CR kind> <CR name> \
    platform.confluent.io/use-old-jetty9=true \
    -n <namespace>
```
Expected: <CR kind>.platform.confluent.io/<CR name> annotated

Note

Without this annotation, you cannot log in to Confluent Platform 7.x components using basic authentication, and you get a login prompt loop in Control Center. For more information, see Issue: Authentication failures when using CFK 3.0 with Confluent Platform 7.x.

Step 2: Disable resource reconciliation

Disable reconciliation for every Day-1 and Day-2 custom resource (CR) kind that you have deployed in each namespace where you have deployed Confluent Platform. This prevents rolling restarts during the CFK upgrade.

Run the following command for each CR, replacing <CR kind> and <CR name> with the resource type and name:

kubectl annotate <CR kind> <CR name> \
    platform.confluent.io/block-reconcile=true \
    -n <namespace>

For example:

kubectl annotate kafka kafka \
    platform.confluent.io/block-reconcile=true \
    -n <namespace>

Expected: <CR kind>.platform.confluent.io/<CR name> annotated

Day-1 CRs include, for example, Connect, ControlCenter, Kafka, KafkaRestProxy, KsqlDB, SchemaRegistry, and either KRaftController for KRaft-based Confluent Platform or Zookeeper for ZooKeeper-based Confluent Platform. Apply the annotation to every Day-1 CR you have deployed, including any kind added in a later CFK version.

Repeat for all deployed Day-1 CRs:

Component	CR kind
Apache Kafka®	`kafka`
Kafka Connect	`connect`
Control Center (Legacy)	`controlcenter`
Confluent REST Proxy	`kafkarestproxy`
ksqlDB	`ksqldb`
Schema Registry	`schemaregistry`
KRaft Controller (KRaft-based only)	`kraftcontroller`
ZooKeeper (ZooKeeper-based only)	`zookeeper`

Day-2 CRs include, for example, KafkaTopic, Schema, SchemaExporter, Connector, ClusterLink, and ConfluentRolebinding. Apply the same annotation to every Day-2 CR you have deployed.

Verify reconciliation is disabled on each CR:

kubectl get <CR kind> <CR name> -n <namespace> \
  -o jsonpath='{.metadata.annotations.platform\.confluent\.io/block-reconcile}'

Expected: true

Step 3: Update the CFK Helm repository

To update the CFK Helm repository, run the following commands:

helm repo add confluentinc https://packages.confluent.io/helm

helm repo update

Expected: Update Complete. ⎈Happy Helming!⎈

Step 4: Pull the CFK chart

Pull the chart using one of the following options.

Latest version

helm pull confluentinc/confluent-for-kubernetes --untar

Specific version

Get the image tag from Confluent for Kubernetes image tags:

helm pull confluentinc/confluent-for-kubernetes --version <cfk_image_tag> --untar

Download bundle

For details, see Deploy CFK using the download bundle.

The --untar flag unpacks the chart into a confluent-for-kubernetes/ directory in your current location. Run Step 5 (and the helm upgrade) from this same directory so the confluent-for-kubernetes/crds/ path resolves.

Expected: a confluent-for-kubernetes/ directory containing crds/, templates/, Chart.yaml, and values.yaml.

Step 5: Upgrade Confluent Platform Custom Resource Definitions (CRDs)

Apply the CRDs from the chart directory. Helm does not upgrade CRDs automatically, so you must do this step manually. For details, see the Helm documentation.

CFK provides the CRDs in two locations:

crds/, at the root of the download bundle. These CRDs include full field descriptions (about 4.7 MB). Use these CRDs, because the descriptions make kubectl explain and CR validation more useful.
confluent-for-kubernetes/crds/, inside the CFK Helm chart. This location is available whether you got the chart from the Helm repository or from the download bundle, but the path differs by source. If you got the chart from the Helm repository, the path is confluent-for-kubernetes/crds/. If you got the chart from the download bundle, the chart is nested one level deeper, so the path is helm/confluent-for-kubernetes/crds/. These CRDs have field descriptions removed (about 1.8 MB) so that Helm can track them within its ConfigMap size limit.

Use server-side apply. It upgrades the CRDs cleanly without the client-side annotation-size limit or the per-CRD last-applied-configuration warnings.

If you have the download bundle, apply the full CRDs from the root-level crds/ directory:

kubectl apply --server-side=true -f crds/

If you only have the Helm chart, apply the description-stripped CRDs from the chart directory instead:

kubectl apply --server-side=true -f confluent-for-kubernetes/crds/

Expected: Each CRD shows serverside-applied.

If you run the kubectl apply command without the --server-side=true flag and get an error similar to the following:

The CustomResourceDefinition "kafkas.platform.confluent.io" is invalid:
metadata.annotations: Too long: must have at most 262144 bytes make: ***
[install-crds] Error 1

Re-run the command with the --server-side=true flag.

If the server-side apply returns a conflict error:

Apply failed with 1 conflict: conflict with "helm" using
apiextensions.k8s.io/v1: .spec.versions

Re-run with --force-conflicts:

kubectl apply --server-side=true --force-conflicts -f confluent-for-kubernetes/crds/

Note

You can also use client-side apply (kubectl apply -f confluent-for-kubernetes/crds/), but it prints a missing last-applied-configuration warning for each CRD (patched automatically) and can fail on large CRDs with metadata.annotations: Too long: must have at most 262144 bytes. Server-side apply avoids both.

Step 6: Upgrade CFK

Upgrade the operator using the option that matches your deployment.

Custom values file

Run the upgrade with --dry-run first to preview the changes. For details about the values file, see Deploy CFK with Custom Values.

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values <path-to-values-file> \
  -n <namespace> \
  --dry-run

After you review the output, apply the upgrade.

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values <path-to-values-file> \
  -n <namespace>

No values file

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  -n <namespace>

Global license

If you use a global license (globalLicense: true), specify the license key on the upgrade command. For details, see Update CFK global license.

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --values values.yaml \
  -n <namespace> \
  --set licenseKey=<cfk_license_key>

Download bundle

Follow the steps in Deploy CFK using the download bundle.

Expected: Release "confluent-operator" has been upgraded. Happy Helming!

After the upgrade completes, verify that the operator pod is healthy before you continue. Resource reconciliation remains blocked, so your Confluent Platform components are not affected yet.

To upgrade to a specific version (hotfix or patch), choose one of the following options:

If you use a custom values file, update the image.tag in values.yaml and then run the helm upgrade command above with --values <path-to-values-file>.
```
image:
  tag: "<cfk_image_tag>"
```

If you do not use a custom values file, pass the target version with --version on the upgrade command.

helm upgrade --install confluent-operator \
  confluentinc/confluent-for-kubernetes \
  --version <cfk_image_tag> \
  -n <namespace>

Step 7: Upgrade CFK init container

Stage this change while resource reconciliation is still blocked. When you enable reconciliation in the next step, a single rolling restart applies both the operator upgrade and the new init container image.

In each Confluent Platform component CR, update the CFK init container image tag to the tag for your target CFK version. The example below uses 3.3.0, which is the canonical tag for the current CFK minor version. For the exact tag for any CFK release (including patch releases), see the Confluent for Kubernetes image tags reference table.

kind: <Confluent component>
spec:
  image:
    init: confluentinc/confluent-init-container:3.3.0

Important

Do not skip this step. Starting in CFK 3.3.0, the init container delivers the cfkprober binary that the new exec liveness probe runs. If you do not update the init container image tag, the older init container does not deliver the cfkprober binary. As a result, the liveness probe fails open and reports the component as live even when it is not. For details, see Liveness probe change to exec in CFK 3.3.0.

Apply the change to each component CR with kubectl patch (or edit the CR directly with kubectl edit). For example, for Kafka:

kubectl patch kafka <CR name> -n <namespace> --type merge \
  -p '{"spec":{"image":{"init":"confluentinc/confluent-init-container:3.3.0"}}}'

Expected: kafka.platform.confluent.io/<CR name> patched

Step 8: Enable resource reconciliation

Re-enable reconciliation for every Day-1 and Day-2 CR that you disabled in Step 2. For each CR, run one of the following commands. The first command sets the annotation to false so CFK can clear the ReconcileIsBlocked status condition. The second command removes the annotation entirely:

kubectl annotate <CR kind> <CR name> \
  platform.confluent.io/block-reconcile=false \
  -n <namespace> --overwrite

OR

kubectl annotate <CR kind> <CR name> \
  platform.confluent.io/block-reconcile- \
  -n <namespace>

Verify the annotation no longer blocks reconciliation:

kubectl get <CR kind> <CR name> -n <namespace> \
  -o jsonpath='{.metadata.annotations.platform\.confluent\.io/block-reconcile}'

Expected: empty output if you removed the annotation, or false if you set it to false.

Note

When reconciliation resumes, the upgraded CFK reconciles each component, applying both the operator upgrade and the staged init container image update in a single rolling restart. You might briefly see pods restart or show Error or CrashLoopBackOff while they reconnect to their dependencies. For example, Schema Registry may crash while Kafka is restarting. The pods recover automatically. Wait for each component CR to return to STATUS: RUNNING before treating the upgrade as failed.

Final verification

After completing all steps, verify the CFK Operator and all Confluent Platform component CRs are healthy on the new version.

Verify the CFK chart and pod version:
```
helm list -n <namespace>
```
Expected: confluent-operator shows the updated chart version.
```
kubectl get pods -n <namespace> | grep confluent-operator
```
Expected: The confluent-operator pod is in Running state.
Verify each Confluent Platform component CR has returned to RUNNING:
```
kubectl get <CR kind> <CR name> -n <namespace>
```
Expected: STATUS column shows RUNNING.
Verify the init container tag actually in use matches your target:
```
kubectl get pod <component-pod> -n <namespace> \
  -o jsonpath='{.spec.initContainers[*].image}'
```
Expected: the image tag matches what you set in Step 7 (for example, confluentinc/confluent-init-container:3.3.0).

Next steps

Now that CFK is upgraded, upgrade your Confluent Platform components. For instructions, see Upgrade Confluent Platform Using Confluent for Kubernetes.