# Upgrade Confluent Platform Using Confluent for Kubernetes Upgrade Confluent Platform components using Confluent for Kubernetes by choosing your upgrade path, preparing your clusters, and running the upgrade steps for KRaft-based or ZooKeeper-based deployments. Before you start the upgrade, review the [pre-upgrade tasks and known issues](co-upgrade-considerations.md#co-upgrade-additional-considerations) so the upgrade is not interrupted. For configurations that cannot be updated on deployed clusters, see [Configurations that can’t change after deployment](co-plan-availability.md#co-update). ## Known issues Review the following known issues before you start the upgrade. #### WARNING Before you update the `cp-server` image tag to Confluent Platform 8.0 or later, remove or update any deprecated broker and topic configuration overrides in your Apache Kafka® CR, KRaftController CR, and KafkaTopic CRs. Leftover values such as `message.format.version` or a `segment.bytes` value below 1MiB cause brokers to fail to start. For details, see [Remove deprecated topic and broker configurations before Confluent Platform 8.0 upgrade](co-upgrade-overview.md#co-upgrade-remediate-deprecated-configs). A separate issue affects OAuth client assertion authentication starting in Confluent Platform 8.3.x. #### NOTE On Confluent Platform 8.3.x, components that authenticate to Kafka using OAuth client assertions (such as Schema Registry, Connect, ksqlDB, Control Center, and REST Proxy) fail to start. This happens because Kafka 4.3 rejects the configured private-key file under an empty, deny-by-default allowlist. Before you upgrade, set the `-Dorg.apache.kafka.sasl.oauthbearer.allowed.files` JVM system property on the affected component. If you configure issuer or JSON Web Key Set (JWKS) endpoint URLs, also set `-Dorg.apache.kafka.sasl.oauthbearer.allowed.urls`. ## Before you begin ### Identify your deployment mode Your upgrade path depends on whether your Kafka cluster runs in KRaft mode or ZooKeeper mode. Check for a `KRaftController` resource in your namespace: ```bash kubectl get kraftcontroller -n ``` - If the command returns a `kraftcontroller` resource, your deployment is KRaft-based. Use the KRaft upgrade path. - If the command returns `No resources found`, your deployment is ZooKeeper-based. Use the ZooKeeper upgrade path. ### Get your current CR files Each upgrade step updates the image tag in a Confluent Platform component CR. If you do not already have the CR YAML files locally, export them from the cluster. For each component you plan to upgrade, save the current CR to a file: ```bash kubectl get -n -o yaml > .yaml ``` For example, to save the Kafka CR: ```bash kubectl get kafka kafka -n -o yaml > kafka.yaml ``` Edit the saved file when a step instructs you to update the image tag, then apply the file with `kubectl apply -f `. ## Choose your upgrade path ![Decision tree for choosing the |kraft| or |zk| upgrade order, including the |c3-short++| 2.0/2.1 exception](images/co-upgrade-choose-path.svg) **If your deployment is KRaft-based (Confluent Platform 7.x or later), upgrade in this order:** 1. [KRaft controllers](#co-upgrade-kraft) 2. [Kafka brokers](#co-upgrade-ak-kraft) 3. [Update metadata version](#co-update-metadata-version) 4. [Other Confluent components](#co-upgrade-other-components) (excluding Control Center and Control Center (Legacy)), in any order 5. [Control Center](#co-upgrade-other-components) **If your deployment is ZooKeeper-based (Confluent Platform 7.9 or earlier only), upgrade in this order:** 1. [ZooKeeper](#co-upgrade-other-components) 2. [Kafka](#co-upgrade-ak) 3. [Other Confluent components](#co-upgrade-other-components) (excluding Control Center and Control Center (Legacy)), in any order 4. [Control Center](#co-upgrade-other-components) ZooKeeper, the other Confluent components, and Control Center all use the same generic procedure in [Upgrade other Confluent Platform components](#co-upgrade-other-components) below: repeat those steps for each component, in the order shown in the preceding list. ## Upgrade a ZooKeeper-based deployment to Confluent Platform 8.0 Confluent Platform 8.0 removes ZooKeeper, so a ZooKeeper-based deployment must migrate to KRaft before upgrading to 8.0. The steps depend on your starting Confluent Platform version. **If you are on Confluent Platform 7.6.x or earlier:** 1. [Upgrade Kafka to 7.9 using the ZooKeeper-based upgrade path](#co-upgrade-cp-steps). Confluent Platform 7.7.0 and later support production-level KRaft migration, so you must first reach a supported version. 2. [Migrate the ZooKeeper-based Confluent Platform to KRaft in 7.9](co-migrate-kraft-procedure.md#co-migrate-kraft). 3. [Upgrade the KRaft-based Confluent Platform to 8.0](#co-upgrade-kraft-based-cp). **If you are on Confluent Platform 7.7.x or later:** 1. [Migrate the ZooKeeper-based Confluent Platform to KRaft in your current version](co-migrate-kraft-procedure.md#co-migrate-kraft). 2. [Upgrade the KRaft-based Confluent Platform to 8.0](#co-upgrade-kraft-based-cp). **If you are upgrading to Confluent Platform 8.0 or later and already have Control Center 2.0 or 2.1 installed, upgrade in this order:** Confluent Platform 8.0 does not work with Control Center 2.0 or 2.1. When upgrading KRaft-based Confluent Platform to 8.0 and Control Center from 2.0 or 2.1 to 2.2, upgrade Control Center *before* upgrading Kafka: 1. [Control Center](#co-upgrade-other-components). For migration steps, see the [Control Center Migration guide](https://docs.confluent.io/control-center/current/installation/overview.html#migration). 2. [KRaft controllers](#co-upgrade-kraft) 3. [Kafka brokers](#co-upgrade-ak-kraft) 4. [Other Confluent components](#co-upgrade-other-components) (excluding Control Center and Control Center (Legacy)), in any order ## Upgrade Kafka and KRaft in the KRaft mode #### WARNING **Known limitation: Upgrade from Confluent Platform 7.7.3 or 7.8.x to 7.9.x** When upgrading a KRaft-based cluster from Confluent Platform 7.7.3 or 7.8.x to 7.9.x, upgrade Kafka brokers *before* KRaft controllers. If a broker restarts while controllers are already on 7.9.x, the broker fails to register and enters a crash loop. ![Decision tree for the broker-before-controller order required when upgrading |cp| 7.7.3 or 7.8.x to 7.9.0 through 7.9.2](images/co-upgrade-broker-controller-order.svg) This occurs because Confluent Platform 7.9.x controllers require brokers to include the `confluent.metadata.version` field during registration. Older brokers remove this field from the registration request because they misidentify the 7.9.x controller as an Kafka controller rather than a Confluent Platform controller. This misidentification causes registration to fail. **To avoid this issue:** * Upgrade directly to Confluent Platform 7.9.3 or later and follow the standard upgrade order. * If upgrading to Confluent Platform 7.9.0–7.9.2, upgrade Kafka brokers first and wait for them to become healthy before upgrading controllers. Avoid broker restarts while controllers are on 7.9.x. Controller logs show this error when the issue occurs: ```text [ControllerServer id=9990] registerBroker: event failed with UnsupportedVersionException ... Exception message: Unable to register broker 0 because it does not support confluent.metadata.version. Please upgrade your broker to Confluent Platform, or downgrade your controller to Apache Kafka. Brokers must be upgraded first in a rolling upgrade from Apache to CP. ``` Clusters starting on 7.7.2 or earlier, and clusters already on 7.9.3 or later, are not affected. With that limitation in mind, review the following guidance for the steps below. #### IMPORTANT **Read the warning above before starting Step 1 of the |kraft| controller and |ak| broker upgrade below.** If you are upgrading from Confluent Platform 7.7.3 or 7.8.x to Confluent Platform 7.9.0, 7.9.1, or 7.9.2, do *Step 2 before Step 1* (Kafka brokers before KRaft controllers). Following Steps 1 → 2 in the order shown causes a broker registration crash loop. For all other upgrade paths (including upgrading directly to Confluent Platform 7.9.3 or later, or to Confluent Platform 8.x), follow Steps 1 → 2 → 3 in order. ### Step 1: Upgrade KRaft controllers If your `KRaftController` CR uses a name other than `kraftcontroller`, replace the pod and statefulset names in the commands below accordingly. 1. In the `KRaftController` custom resource (CR), update the image tag to the target Confluent Platform version: ```yaml kind: KRaftController spec: image: application: confluentinc/cp-server:8.3.0 ``` 2. In the same CR, verify that the init container image tag is the current CFK version: ```yaml spec: image: init: confluentinc/confluent-init-container:3.3.0 ``` 3. Apply the updated CR: ```bash kubectl apply -f -n ``` Expected: `kraftcontroller.platform.confluent.io/kraftcontroller configured` 4. Wait for the rolling restart to complete: ```bash kubectl rollout status statefulset/kraftcontroller -n ``` Expected: `statefulset rolling update complete`. 5. Verify the KRaft controllers are running on the target version: ```bash kubectl get kraftcontroller -n ``` Expected: STATUS shows `RUNNING`. ```bash kubectl get pod kraftcontroller-0 -n \ -o jsonpath='{.spec.containers[*].image}' ``` Expected: image tag matches `confluentinc/cp-server:8.3.0`. ### Step 2: Upgrade Kafka brokers 1. In the `Kafka` CR, update the image tag to the target Confluent Platform version: ```yaml kind: Kafka spec: image: application: confluentinc/cp-server:8.3.0 ``` 2. In the same CR, verify that the init container image tag is the current CFK version: ```yaml spec: image: init: confluentinc/confluent-init-container:3.3.0 ``` 3. Apply the updated CR: ```bash kubectl apply -f -n ``` Expected: `kafka.platform.confluent.io/kafka configured` 4. Wait for the rolling restart to complete: ```bash kubectl rollout status statefulset/kafka -n ``` Expected: `statefulset rolling update complete`. 5. Verify the Kafka brokers are running on the target version: ```bash kubectl get kafka -n ``` Expected: STATUS shows `RUNNING`. ```bash kubectl get pod kafka-0 -n \ -o jsonpath='{.spec.containers[*].image}' ``` Expected: image tag matches `confluentinc/cp-server:8.3.0`. ### Step 3: Update metadata version After verifying cluster health and performance, increment the metadata version for controllers and brokers. Run `kafka-features` from inside a Kafka broker pod. The binary is preinstalled, and the pod can reach the internal bootstrap listener (`kafka:9071` by default): ```bash kubectl exec kafka-0 -n -- \ kafka-features --bootstrap-server kafka:9071 \ upgrade \ --metadata 4.3 ``` Expected: `confluent.metadata.version was upgraded to .` The numeric is an internal version code. You can see the effective version using the `describe` command shown below. If your Kafka CR uses a name other than `kafka`, replace the pod name (`kafka-0`) and the bootstrap host accordingly. The `--metadata` value must match the Apache Kafka® version that ships with your target Confluent Platform release. The example above uses `4.3` (the value for Confluent Platform 8.3.0). For other Confluent Platform versions, use: | Target Confluent Platform version | `--metadata` value | |-------------------------------------|----------------------| | 7.6 | `3.6` | | 7.7 | `3.7` | | 7.8 | `3.8` | | 7.9 | `3.9` | | 8 | `4.0` | | 8.1 | `4.1` | | 8.2 | `4.2` | | 8.3 | `4.3` | Verify the metadata version: ```bash kubectl exec kafka-0 -n -- \ kafka-features --bootstrap-server kafka:9071 describe ``` Expected: The `confluent.metadata.version` row shows a `FinalizedVersionLevel` matching your target (for example, `CP-7.9-IV0B` after setting `--metadata 3.9`). #### WARNING Do not use the `--release-version` option with the `kafka-features` command on clusters managed by CFK. This option upgrades `kraft.version` from `0` to `1`, which is not part of the Confluent Platform upgrade process. CFK does not support upgrading `kraft.version`. Use only the `--metadata` flag for CFK-managed clusters. For more information about the metadata version, see the Confluent Platform documentation: [Upgrade Kafka brokers in KRaft mode](https://docs.confluent.io/platform/current/installation/upgrade.html#steps-for-upgrading-to-version-x-kraft-mode). ## Upgrade Kafka in the ZooKeeper mode This section only applies to Confluent Platform 7.9 or earlier. #### IMPORTANT Starting with Confluent Platform version 8.0, ZooKeeper is no longer part of Confluent Platform. You have two options to upgrade Kafka: * [Automatic upgrade](#co-upgrade-ak-auto) (recommended, CFK 2.8 or later) * [Manual upgrade](#co-upgrade-ak-manual) (CFK 2.7 or earlier, or for manual control) ### Automatic upgrade of Kafka Starting in CFK 2.8, CFK automates the Kafka upgrade process by automatically managing `inter.broker.protocol.version` and `log.message.format.version` during upgrades. CFK enables this upgrade option by default. It applies to: * Standard Confluent Platform Docker images only (not custom image registries) * ZooKeeper-based Kafka clusters (Confluent Platform 7.9 or earlier only) #### NOTE CFK sets `inter.broker.protocol.version` in CP format. For example, when upgrading from Confluent Platform 7.3 to 7.6, CFK sets it as `CP-7.3` rather than the Kafka format `3.3`. #### Step 1: (Optional) Override the default inter-broker version CFK defaults to inter-broker version `2.6`. To override this before running the auto upgrade (for example, when upgrading from Confluent Platform 7.3 to 7.6), set the bootstrap inter-broker protocol (IBP) version annotation: ```bash kubectl annotate kafka kafka \ platform.confluent.io/bootstrap-ibp-version="" \ -n ``` Set `` in CFK format. For example, use `CP-7.3` when upgrading from Confluent Platform 7.3. Verify the update: ```bash kubectl get kafka kafka -n -o yaml | grep previousClusterVersion ``` #### NOTE Use this override only for a cluster that has already completed at least one automatic upgrade, so `previousClusterVersion` reflects a real prior Confluent Platform version. If this is the first automatic upgrade since the cluster was originally deployed, `previousClusterVersion` does not yet reflect a genuine prior version. As a result, the value you see here may not match what you set. In that case, this step does not apply to you: skip the override and continue to Step 2. Remove the annotation after verifying: ```bash kubectl annotate kafka kafka \ platform.confluent.io/bootstrap-ibp-version- \ -n ``` #### Step 2: Verify the init container image tag In the `Kafka` CR, verify that the init container image tag is the current CFK version: ```yaml spec: image: init: confluentinc/confluent-init-container:3.3.0 ``` #### Step 3: Update the Kafka image tag In the `Kafka` CR, update the image tag to the target Confluent Platform version: ```yaml spec: image: application: confluentinc/cp-server:8.3.0 ``` #### Step 4: Apply the `Kafka` CR The automatic upgrade triggers when the image tag is later than `status.currentClusterVersion`: ```bash kubectl apply -f -n ``` Expected: `kafka.platform.confluent.io/kafka configured` #### Step 5: Verify the final cluster versions To verify the final cluster versions, run the following command: ```bash kubectl get kafka kafka -n -o yaml | grep previousClusterVersion ``` ```bash kubectl get kafka kafka -n -o yaml | grep currentClusterVersion ``` Expected: For example, after upgrading from 7.3 to 7.6: ```yaml status.previousClusterVersion: 7.3 status.currentClusterVersion: 7.6 ``` #### NOTE CFK does not set `previousClusterVersion` and `currentClusterVersion` when you [manually upgrade Kafka](#co-upgrade-ak-manual). The automatic upgrade always sets `inter.broker.protocol.version` to the previous Kafka version. In the above example, CFK sets `inter.broker.protocol.version` to `3.3`. On the next upgrade from Confluent Platform 7.6 to 7.7, CFK sets it to `3.6`. ### Manual upgrade of Kafka Use this method if you are on CFK 2.7 or earlier, or if you want manual control over the upgrade process. #### Step 1: Update the Kafka image tag In the `Kafka` CR, update the image tag to the target Confluent Platform version: ```yaml spec: image: application: confluentinc/cp-server:8.3.0 ``` #### Step 2: Verify the init container image tag Verify that the init container image tag is the current CFK version: ```yaml spec: image: init: confluentinc/confluent-init-container:3.3.0 ``` #### Step 3: Disable the automatic upgrade feature Add the following annotation to the `Kafka` CR to disable the automatic upgrade feature: ```yaml metadata: annotations: platform.confluent.io/automatic-ibp-upgrade: "disable" ``` #### Step 4: Set the current inter-broker protocol version Set `inter.broker.protocol.version` to your current Confluent Platform version. Look up the protocol version for your current Confluent Platform release in the [Kafka upgrade documentation](https://docs.confluent.io/platform/current/installation/upgrade.html#:~:text=The%20following%20table%20shows%20the%20broker%20protocol%20and%20log%20message%20format%20versions%20that%20correspond%20with%20each%20Confluent%20Platform%20version). If upgrading from Confluent Platform 6.2.x or earlier, also set `log.message.format.version`. If upgrading from Confluent Platform 7.0 or later, set only `inter.broker.protocol.version`. For example, when upgrading from Confluent Platform 7.4.1: ```yaml spec: configOverrides: server: - inter.broker.protocol.version=3.4 ``` #### Step 5: Apply the `Kafka` CR Apply the updated CR: ```bash kubectl apply -f -n ``` Expected: `kafka.platform.confluent.io/kafka configured` #### Step 6: Set the target inter-broker protocol version After the brokers have upgraded, set `inter.broker.protocol.version` to the target Confluent Platform version. Look up the target protocol version in the [Kafka upgrade documentation](https://docs.confluent.io/platform/current/installation/upgrade.html#:~:text=The%20following%20table%20shows%20the%20broker%20protocol%20and%20log%20message%20format%20versions%20that%20correspond%20with%20each%20Confluent%20Platform%20version) (the same table linked in Step 4). For example, when upgrading to 8.3, the target version is `4.3`: ```yaml spec: configOverrides: server: - inter.broker.protocol.version= ``` #### Step 7: Apply the `Kafka` CR again Apply the CR again: ```bash kubectl apply -f -n ``` Expected: `kafka.platform.confluent.io/kafka configured` #### NOTE CFK does not set the `previousClusterVersion` and `currentClusterVersion` properties in manual upgrades. These properties do not affect your cluster, but you can edit them manually if needed. ## Upgrade other Confluent Platform components Upgrade Confluent Platform components including ZooKeeper (Confluent Platform 7.9 and earlier) as follows: 1. **Update the component image tag.** In the component CR, set the image tag to the target Confluent Platform version: ```yaml spec: image: application: :8.3.0 ``` 2. **Verify the init container image tag matches your Confluent for Kubernetes version.** In the same CR, confirm the init container tag is set to your current CFK version (the same tag set in [Step 7: Upgrade CFK init container](co-upgrade-cfk.md#co-upgrade-init-container)): ```yaml spec: image: init: confluentinc/confluent-init-container:3.3.0 ``` 3. If upgrading Control Center, also update the Prometheus and Alertmanager image tags. Control Center uses independent versions, not Confluent Platform release versions. Set `` to the Control Center release: ```yaml kind: ControlCenter spec: image: application: confluentinc/cp-enterprise-control-center-next-gen: init: confluentinc/confluent-init-container:3.3.0 services: prometheus: image: confluentinc/cp-enterprise-prometheus: pvc: dataVolumeCapacity: 10Gi alertmanager: image: confluentinc/cp-enterprise-alertmanager: ``` Set `dataVolumeCapacity` to a capacity suitable for your environment. `10Gi` is shown here only as an example. 4. If upgrading Control Center (Legacy) and you migrated from Operator 1.x to CFK 2.3.0 or earlier, and are now upgrading to CFK 2.4.0 or later with Confluent Platform 7.1 or earlier, set the following property: #### IMPORTANT Starting with Confluent Platform version 8.0, Confluent Control Center (Legacy) is no longer supported with Confluent Platform. Use Control Center with Confluent Platform 8.0 and later. ```yaml spec: configOverrides: server: - confluent.controlcenter.internal.topics.partitions=4 ``` 5. Apply the component CR: ```bash kubectl apply -f -n ``` Expected: `.platform.confluent.io/ configured` 6. Verify the component is running: ```bash kubectl get -n ``` Expected: STATUS shows `RUNNING`. ## Next steps After your Confluent Platform components are upgraded: - If you are migrating from Confluent Control Center (Legacy) to Control Center, or upgrading Control Center 2.0 or 2.1 to 2.2, see [Migrate to and Upgrade Control Center](co-upgrade-c3.md#co-upgrade-c3). - For component-specific post-upgrade tasks and known issues, see [Pre-upgrade tasks and known issues](co-upgrade-considerations.md#co-upgrade-additional-considerations).