Confluent Developer Marketplace Get started
  1. Home
  2. Platform
  3. CFK
  4. Upgrade Confluent Platform Using Confluent for Kubernetes

Upgrade Confluent Platform Using Confluent for Kubernetes

Upgrade a Confluent for Kubernetes (CFK) deployment using the following process.

In this section

Before you start, review Pre-upgrade tasks and known issues.

Why should you upgrade

The following are some of the common reasons to upgrade:

  • To pick up security fixes, bug fixes, and new features in CFK and Confluent Platform.

  • To move to a CFK version that supports a newer Confluent Platform version.

  • To prepare for the architectural changes in Confluent Platform 8.0, which is the removal of ZooKeeper and the move to Control Center. For details, see the conditions in Before you choose your upgrade path section.

What you’re upgrading

A CFK deployment has two independent layers that you upgrade separately:

  1. You install and upgrade the CFK Operator with Helm. The Operator watches your Confluent Platform custom resources (CRs) and reconciles cluster state. The Operator package also includes the Custom Resource Definitions (CRDs) and the init container image.

  2. The Confluent Platform components are Apache Kafka®, KRaft controllers, Schema Registry, Connect, ksqlDB, Control Center, and REST Proxy. Each runs from a container image whose tag is set in its CR.

Check your current versions

Verify your Kubernetes cluster is in the supported environments for the target CFK version, then identify your current CFK and Confluent Platform versions:

  • Check your current CFK version:

    helm list -n <namespace>

    Expected: confluent-operator with your current CFK version.

  • Check your current Confluent Platform version:

    kubectl get kafka <kafka-name> -n <namespace> -o jsonpath='{.status.currentClusterVersion}'

    Expected: Your current Confluent Platform version in major.minor format, for example 7.9.

Before you choose your upgrade path

Review the following conditions before you choose your upgrade path. For pre-upgrade tasks and known issues, see Pre-upgrade tasks and known issues.

Tip

Always check the support matrix before you plan an upgrade. It is the source of truth for which Confluent Platform versions each CFK version supports, and confirms whether your starting and target combinations are valid.

CFK and Confluent Platform compatibility

Starting in CFK 2.9, CFK supports all supported Confluent Platform versions.

For CFK versions earlier than 2.9, a CFK and Confluent Platform combination not listed in the support matrix is still supported until you reach the final state where both are upgraded to a supported combination.

For example, if you upgrade CFK from 2.6.1 to 3.3.0 along with Confluent Platform from 7.3.1 to 8.3.0, the combination of CFK 3.3.0 and Confluent Platform 7.3.1 is temporarily supported during the upgrade.

Liveness probe change to exec in CFK 3.3.0

Starting in CFK 3.3.0, the liveness probe for the components that always run a liveness probe (Connect, Schema Registry, ksqlDB, Control Center, REST Proxy, Unified Stream Manager Agent, and Confluent Gateway) changes from an HTTP (httpGet) or TCP (tcpSocket) probe to an exec probe that runs the bundled cfkprober binary. After you upgrade, if you inspect a pod spec and see an exec probe in place of the previous httpGet or tcpSocket probe, this is expected and not a misconfiguration. The upgrade steps do not change.

For Kafka and KRaft, the liveness probe remains opt-in, and when enabled, it also runs as an exec probe now. The probe runs in one of two modes:

  • By default, the probe runs the cfkprober binary, as the other components do.

  • If you set the useProcNetPortCheck flag, the probe reads the /proc/net/tcp file to find the port instead of communicating with the server. This helps avoid Transport Layer Security (TLS)-related errors. This flag is meaningful only for the Kafka and KRaft custom resources, where it avoids the TLS handshake issues described in the preceding section.

The CFK init container delivers the cfkprober binary to the pod. When you upgrade, you must also update the init container image tag as described in the CFK upgrade process. If you skip the init container upgrade, the older init container does not deliver the cfkprober binary, so the new exec liveness probe fails open and always reports the component as live.

Add RBAC permissions for custom roles before upgrading to 3.3

If you manage the CFK ServiceAccount permissions manually instead of letting Helm create them, add the following rules to your Role or ClusterRole before you upgrade to CFK 3.3 or later. Without these permissions, the CFK pod repeatedly crashes with an error like:

Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is
forbidden: User "system:serviceaccount:<namespace>:confluent-for-
kubernetes" cannot list resource "jobs" in API group "batch" in
the namespace "<namespace>"

- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - get

For the full set of permissions required by CFK, see the reference role and role binding examples linked from Configure Kubernetes RBAC and Custom Resource Definitions.

ZooKeeper removal in Confluent Platform 8.0

Confluent Platform 8.0 removes ZooKeeper. Migrate your ZooKeeper-based deployment to KRaft before upgrading to 8.0. For migration steps, see Migrate a Single Cluster from ZooKeeper to KRaft.

Control Center replaces Control Center (Legacy) in Confluent Platform 8.0

Confluent Platform 8.0 or later doesn’t support Control Center (Legacy). Before you upgrade to Confluent Platform 8.x, migrate your alerts from Control Center (Legacy) to Control Center, and switch to Control Center. For migration steps, see Migrate to and Upgrade Control Center.

Choose your upgrade path

Choose your path based on which layer you want to change. The CFK Operator, the Confluent Platform components, or both. Upgrading both is the most common case and keeps your CFK and Confluent Platform versions aligned with the support matrix.

|co| Operator and |cp| components upgrade independently, with three supported upgrade paths

CFK Operator and Confluent Platform components upgrade independently, with three supported upgrade paths

Upgrade Confluent for Kubernetes only:

Make sure the CFK version you are upgrading to supports your current Confluent Platform version. If it does not, upgrade Confluent Platform into the new CFK’s supported range first. For instructions, see Upgrade Confluent for Kubernetes.

Upgrade Confluent Platform only:

Make sure your current CFK version supports both the starting and ending Confluent Platform versions. For instructions, see Upgrade Confluent Platform Using Confluent for Kubernetes.

Upgrade both Confluent for Kubernetes and Confluent Platform:

If your target Confluent Platform version is below 8.0, complete only steps 1 and 2 below. Steps 3 and 4 apply only when your target is Confluent Platform 8.0 or later.

  1. Upgrade Confluent for Kubernetes

  2. Upgrade Confluent Platform Using Confluent for Kubernetes

  3. (Confluent Platform 8.0 or later only.) If your deployment is still ZooKeeper-based, migrate to KRaft in your current version.

  4. (Confluent Platform 8.0 or later only.) If you completed the preceding step, or your deployment was already KRaft-based, upgrade to Confluent Platform 8.0.

To upgrade from Confluent Operator 1.x to Confluent for Kubernetes, see Migrate to Confluent for Kubernetes.

Remove deprecated topic and broker configurations before Confluent Platform 8.0 upgrade

Confluent Platform 8.0 removes or raises the minimum value of several broker and topic configurations. If your existing Kafka CR, KRaftController CR, or KafkaTopic CRs override these configurations with values that are no longer accepted, the brokers fail to start with errors similar to:

Invalid value 1000000 for configuration segment.bytes: Value must be
at least 1048576

or

Unknown topic config name: message.timestamp.difference.max.ms

Before you update the cp-server image tag to Confluent Platform 8.0 or later, update or remove the overrides described below in your Kafka CR, KRaftController CR, and KafkaTopic CRs, and any topic-level dynamic configurations that you set with the kafka-configs tool.

Increase segment size to at least 1MiB

Starting in Confluent Platform 8.0, KIP-1030 raises the minimum value of segment.bytes (topic-level) and log.segment.bytes (broker-level) from 14 bytes to 1048576 bytes (1MiB). The default value (1GiB) is unchanged. Topics or brokers configured below 1MiB cause broker startup to fail on Confluent Platform 8.0 and later.

Update broker-level overrides in your Kafka CR:

kind: Kafka
spec:
  configOverrides:
    server:
      - log.segment.bytes=1048576

Update topic-level overrides in your KafkaTopic CRs:

kind: KafkaTopic
spec:
  configs:
    segment.bytes: "1048576"

To find any topics with non-default segment.bytes overrides on a running cluster, run the kafka-configs tool against the Kafka brokers and update any value below 1048576 bytes.

Remove deprecated broker and topic configurations

Remove the following configurations. Leftover values prevent dynamic configuration updates and can return Unknown topic config name errors when CRs are reconciled:

  • message.format.version

  • message.downconversion.enable

  • message.timestamp.difference.max.ms

Next steps