Upgrade Confluent Platform Using Confluent for Kubernetes

Upgrade a Confluent for Kubernetes (CFK) deployment using the following process.

In this section

Example of a multi-version upgrade: See the upgrade path and reasoning applied to a concrete scenario.
Upgrade CFK: Upgrade the CFK Operator using Helm.
Upgrade Confluent Platform: Upgrade Confluent Platform components for KRaft-based or ZooKeeper-based deployments.
Migrate to and upgrade Control Center: Migrate Confluent Control Center (Legacy) alerts to Control Center and upgrade Control Center when moving to Confluent Platform 8.0.

Before you start, review Pre-upgrade tasks and known issues.

Why should you upgrade

The following are some of the common reasons to upgrade:

To pick up security fixes, bug fixes, and new features in CFK and Confluent Platform.
To move to a CFK version that supports a newer Confluent Platform version.
To prepare for the architectural changes in Confluent Platform 8.0, which is the removal of ZooKeeper and the move to Control Center. For details, see the conditions in Before you choose your upgrade path section.

What you’re upgrading

A CFK deployment has two independent layers that you upgrade separately:

You install and upgrade the CFK Operator with Helm. The Operator watches your Confluent Platform custom resources (CRs) and reconciles cluster state. The Operator package also includes the Custom Resource Definitions (CRDs) and the init container image.
The Confluent Platform components are Apache Kafka®, KRaft controllers, Schema Registry, Connect, ksqlDB, Control Center, and REST Proxy. Each runs from a container image whose tag is set in its CR.

Check your current versions

Verify your Kubernetes cluster is in the supported environments for the target CFK version, then identify your current CFK and Confluent Platform versions:

Check your current CFK version:
```
helm list -n <namespace>
```
Expected: confluent-operator with your current CFK version.
Check your current Confluent Platform version:
```
kubectl get kafka <kafka-name> -n <namespace> -o jsonpath='{.status.currentClusterVersion}'
```
Expected: Your current Confluent Platform version in major.minor format, for example 7.9.

Before you choose your upgrade path

Review the following conditions before you choose your upgrade path. For pre-upgrade tasks and known issues, see Pre-upgrade tasks and known issues.

Tip

Always check the support matrix before you plan an upgrade. It is the source of truth for which Confluent Platform versions each CFK version supports, and confirms whether your starting and target combinations are valid.

CFK and Confluent Platform compatibility

Starting in CFK 2.9, CFK supports all supported Confluent Platform versions.

For CFK versions earlier than 2.9, a CFK and Confluent Platform combination not listed in the support matrix is still supported until you reach the final state where both are upgraded to a supported combination.

For example, if you upgrade CFK from 2.6.1 to 3.4.0 along with Confluent Platform from 7.3.1 to 8.3.0, the combination of CFK 3.4.0 and Confluent Platform 7.3.1 is temporarily supported during the upgrade.

Liveness probe change to exec in CFK 3.4.0

Starting in CFK 3.4.0, the liveness probe for the components that always run a liveness probe (Connect, Schema Registry, ksqlDB, Control Center, REST Proxy, Unified Stream Manager Agent, and Confluent Gateway) changes from an HTTP (httpGet) or TCP (tcpSocket) probe to an exec probe that runs the bundled cfkprober binary. After you upgrade, if you inspect a pod spec and see an exec probe in place of the previous httpGet or tcpSocket probe, this is expected and not a misconfiguration. The upgrade steps do not change.

For Kafka and KRaft, the liveness probe remains opt-in, and when enabled, it also runs as an exec probe now. The probe runs in one of two modes:

By default, the probe runs the cfkprober binary, as the other components do.
If you set the useProcNetPortCheck flag, the probe reads the /proc/net/tcp file to find the port instead of communicating with the server. This helps avoid Transport Layer Security (TLS)-related errors. This flag is meaningful only for the Kafka and KRaft custom resources, where it avoids the TLS handshake issues described in the preceding section.

The CFK init container delivers the cfkprober binary to the pod. When you upgrade, you must also update the init container image tag as described in the CFK upgrade process. If you skip the init container upgrade, the older init container does not deliver the cfkprober binary, so the new exec liveness probe fails open and always reports the component as live.

Kafka broker and KRaftController heap-size default change in CFK 3.4.0

Starting in CFK 3.4.0, the default JVM heap size for Kafka brokers and KRaft controllers is capped instead of being set to 100% of the pod memory request:

Kafka broker: -Xms/-Xmx = min(50% of requests.memory, 32Gi)
KRaft controller: -Xms/-Xmx = min(80% of requests.memory, 32Gi)

This applies to both new and existing clusters and triggers a one-time rolling restart of the affected components on upgrade. Components that pin heap explicitly through spec.configOverrides.jvm are unaffected.

If a broker or controller’s live heap usage is close to or above the new cap, reducing its heap on upgrade can cause it to run out of memory. Before upgrading, check whether any broker or controller needs more heap than the new default provides, and if so, pin an explicit value:

spec:
  configOverrides:
    jvm:
      - "-Xms6G"
      - "-Xmx6G"

All other Confluent Platform components are not affected by this change and continue to default to 100% of the pod memory request. For more information, see Default JVM heap size.

Add RBAC permissions for custom roles before upgrading to 3.3

If you manage the CFK ServiceAccount permissions manually instead of letting Helm create them, add the following rules to your Role or ClusterRole before you upgrade to CFK 3.3 or later. Without these permissions, the CFK pod repeatedly crashes with an error like:

Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is
forbidden: User "system:serviceaccount:<namespace>:confluent-for-
kubernetes" cannot list resource "jobs" in API group "batch" in
the namespace "<namespace>"

- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - get

For the full set of permissions required by CFK, see the reference role and role binding examples linked from Configure Kubernetes RBAC and Custom Resource Definitions.

Add RBAC permissions for custom roles before upgrading to 3.4

Starting in CFK 3.4.0, the operator manages its own CRDs by default (see the “Upgrade CRDs” step in Upgrade CFK). This requires the CFK ServiceAccount to have get, list, and watch permission on customresourcedefinitions, plus create, update, and patch permission unless you set manageCRDs: false. This permission is cluster-scoped, because CRDs are cluster-scoped resources, even if you run CFK in namespaced mode.

If you manage the CFK ServiceAccount permissions manually instead of letting Helm create them (rbac: false), add the following rule to your ClusterRole. Apply this rule before you upgrade to CFK 3.4 or later:

- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch

Without get/list on customresourcedefinitions, the operator cannot check the installed CRDs against its own version at all. It logs a warning once and continues to reconcile Confluent Platform components normally.
With get/list/watch but without create/update/patch, the operator checks the installed CRDs against its own version at startup. If they are outdated, it logs a warning and raises a CRDsOutdatedNotManaged event on its own Deployment, but leaves the CRDs untouched. You must then continue to upgrade CRDs manually as described in Upgrade CFK.

Grant the full rule, including create, update, and patch, to get automatic CRD upgrades. This lets you skip the manual step for applying CRDs on future upgrades.

For the full set of permissions required by CFK, see the reference role and role binding examples linked from Configure Kubernetes RBAC and Custom Resource Definitions.

Update Confluent Gateway security settings before you upgrade to 3.4

CFK 3.4 changes how you configure route authentication in the Confluent Gateway custom resource (CR). A Confluent Gateway CR that uses the CFK 3.3 settings isn’t compatible with CFK 3.4. Route authentication is an Early Access feature that is part of topic-based routing and fencing.

If you upgrade only the Confluent Gateway image to 1.4, and keep CFK 3.3, your existing Confluent Gateway CRs keep working.
If you upgrade CFK to 3.4, CFK doesn’t reconcile your existing Confluent Gateway deployments, and the next kubectl apply of a CR that uses the 3.3 settings fails.

Before you upgrade CFK, update each Confluent Gateway CR to the 3.4 settings. In 3.4, you configure security.client once for the route, and you configure security.cluster as a map with one entry for each streaming domain that the route is bound to. Name each entry after its streaming domain. A bound streaming domain with no entry uses identity passthrough, so for a route that used security.auth: passthrough, you can omit security.cluster.

CFK 3.3	CFK 3.4
`security.auth`	`security.cluster.<streaming-domain>.auth`
`security.secretStore`	`security.cluster.<streaming-domain>.secretStore`
`security.cluster.authentication`	`security.cluster.<streaming-domain>.authentication`
`security.cluster.extensionHeaders`	`security.cluster.<streaming-domain>.extensionHeaders`
`security.client`	`security.client` (no change)

The following examples show only the security settings of a route that swaps credentials for one streaming domain, payments.

CFK 3.3:

kind: Gateway
spec:
  routes:
    - name: payments-route
      security:
        auth: swap
        secretStore: my-secret-store
        client:
          authentication:
            type: plain
            jaasConfigPassThrough:
              secretRef: client-credentials
          tls:
            secretRef: gateway-tls
        cluster:
          authentication:
            type: oauth
            jaasConfigPassThrough:
              secretRef: oauth-jaas
            oauthSettings:
              tokenEndpointUri: "https://idp.example.com/oauth/token"

CFK 3.4:

kind: Gateway
spec:
  routes:
    - name: payments-route
      security:
        client:
          authentication:
            type: plain
            jaasConfigPassThrough:
              secretRef: client-credentials
          tls:
            secretRef: gateway-tls
        cluster:
          payments:
            auth: swap
            secretStore: my-secret-store
            authentication:
              type: oauth
              jaasConfigPassThrough:
                secretRef: oauth-jaas
              oauthSettings:
                tokenEndpointUri: "https://idp.example.com/oauth/token"

For all the settings, see Configure authentication for Confluent Gateway.

ZooKeeper removal in Confluent Platform 8.0

Confluent Platform 8.0 removes ZooKeeper. Migrate your ZooKeeper-based deployment to KRaft before upgrading to 8.0. For migration steps, see Migrate a Single Cluster from ZooKeeper to KRaft.

Control Center replaces Control Center (Legacy) in Confluent Platform 8.0

Confluent Platform 8.0 or later doesn’t support Control Center (Legacy). Before you upgrade to Confluent Platform 8.x, migrate your alerts from Control Center (Legacy) to Control Center, and switch to Control Center. For migration steps, see Migrate to and Upgrade Control Center.

Choose your upgrade path

Choose your path based on which layer you want to change. The CFK Operator, the Confluent Platform components, or both. Upgrading both is the most common case and keeps your CFK and Confluent Platform versions aligned with the support matrix.

|co| Operator and |cp| components upgrade independently, with three supported upgrade paths — CFK Operator and Confluent Platform components upgrade independently, with three supported upgrade paths

Upgrade Confluent for Kubernetes only:

Make sure the CFK version you are upgrading to supports your current Confluent Platform version. If it does not, upgrade Confluent Platform into the new CFK’s supported range first. For instructions, see Upgrade Confluent for Kubernetes.

Upgrade Confluent Platform only:

Make sure your current CFK version supports both the starting and ending Confluent Platform versions. For instructions, see Upgrade Confluent Platform Using Confluent for Kubernetes.

Upgrade both Confluent for Kubernetes and Confluent Platform:

If your target Confluent Platform version is below 8.0, complete only steps 1 and 2 below. Steps 3 and 4 apply only when your target is Confluent Platform 8.0 or later.

Upgrade Confluent for Kubernetes
Upgrade Confluent Platform Using Confluent for Kubernetes
(Confluent Platform 8.0 or later only.) If your deployment is still ZooKeeper-based, migrate to KRaft in your current version.
(Confluent Platform 8.0 or later only.) If you completed the preceding step, or your deployment was already KRaft-based, upgrade to Confluent Platform 8.0.

To upgrade from Confluent Operator 1.x to Confluent for Kubernetes, see Migrate to Confluent for Kubernetes.

Remove deprecated topic and broker configurations before Confluent Platform 8.0 upgrade

Confluent Platform 8.0 removes or raises the minimum value of several broker and topic configurations. If your existing Kafka CR, KRaftController CR, or KafkaTopic CRs override these configurations with values that are no longer accepted, the brokers fail to start with errors similar to:

Invalid value 1000000 for configuration segment.bytes: Value must be
at least 1048576

or

Unknown topic config name: message.timestamp.difference.max.ms

Before you update the cp-server image tag to Confluent Platform 8.0 or later, update or remove the overrides described below in your Kafka CR, KRaftController CR, and KafkaTopic CRs, and any topic-level dynamic configurations that you set with the kafka-configs tool.

Increase segment size to at least 1MiB

Starting in Confluent Platform 8.0, KIP-1030 raises the minimum value of segment.bytes (topic-level) and log.segment.bytes (broker-level) from 14 bytes to 1048576 bytes (1MiB). The default value (1GiB) is unchanged. Topics or brokers configured below 1MiB cause broker startup to fail on Confluent Platform 8.0 and later.

Update broker-level overrides in your Kafka CR:

kind: Kafka
spec:
  configOverrides:
    server:
      - log.segment.bytes=1048576

Update topic-level overrides in your KafkaTopic CRs:

kind: KafkaTopic
spec:
  configs:
    segment.bytes: "1048576"

To find any topics with non-default segment.bytes overrides on a running cluster, run the kafka-configs tool against the Kafka brokers and update any value below 1048576 bytes.

Remove deprecated broker and topic configurations

Remove the following configurations. Leftover values prevent dynamic configuration updates and can return Unknown topic config name errors when CRs are reconciled:

message.format.version
message.downconversion.enable
message.timestamp.difference.max.ms

Next steps

See a worked example of a multi-version upgrade, then try the same reasoning against your own starting and target versions.
For pre-upgrade tasks and known issues, see Pre-upgrade tasks and known issues.
To upgrade the CFK Operator, see Upgrade Confluent for Kubernetes.
To upgrade Confluent Platform components, see Upgrade Confluent Platform Using Confluent for Kubernetes.
To migrate Confluent Control Center (Legacy) alerts to Control Center and upgrade Control Center, see Migrate to and Upgrade Control Center.