Pre-upgrade tasks and known issues

Complete the following tasks before you start the upgrade. Review the known issues to expect during the upgrade.

Does this apply to me?

Use the following table to find the sections you need to read for your upgrade.

Pre-upgrade tasks

Deployment scenario	Applicable sections
Upgrading any Apache Kafka® cluster	Resolve under-replicated partitions Avoid simultaneous changes
Upgrading Confluent Platform from 7.x to 8.x	Upgrade Log4j to Log4j 2
Using CFK 3.x with Confluent Platform 7.x, with basic authentication enabled on Schema Registry, Connect, ksqlDB, Control Center (Legacy), or REST Proxy	Basic authentication with JAAS when using CFK 3.x with Confluent Platform 7.x
Using CFK 3.x with Confluent Platform 7.x	Upgrade Log4j to Log4j 2
Upgrading from CFK 2.3.x or earlier with auto-generated certificates	Upgrade auto-generated certificates to configurable auto-generated certificates

Known issues during upgrade

Deployment scenario	Applicable sections
Upgrading Confluent Platform from 7.x to 8.x with Control Center or other telemetry reporters	Disabled JMX reporter when using telemetry reporter in Confluent Platform 8.0.0
Upgrading to CFK 2.9 or later, with Connect deployed	Connect pods restart when upgrading to CFK 2.9 and later
Upgrading Kafka clusters used in Cluster Linking	Kafka clusters involved in Cluster Linking
Running CFK 3.x with Confluent Platform 7.x and upgrading Control Center to 2.4.x or 2.5.x	Control Center crash loop with the Log4j compatibility annotation

Pre-upgrade tasks

Complete the applicable tasks in this section before you start the upgrade.

Avoid simultaneous changes

Upgrade components separately. Do not make configuration changes during a version upgrade. For example, do not change spec.replicas, spec.resources, or spec.configOverrides in the same edit that bumps spec.image.application.

Warning

Do not upgrade the Confluent Platform server image and update configuration at the same time. Doing both at once can stop your upgrade.

Before you start an upgrade:

Verify all configurations are stable and tested.
Verify cluster health.
Back up your current configuration state.

Resolve under-replicated partitions

Check for under-replicated partitions (URPs) on the existing cluster and resolve them before starting the upgrade. An upgrade with URPs present can stall or fail.

Check for URPs using one of the following methods:

Using Control Center (Legacy): Navigate to Overview and check Under replicated partitions. Expected: shows 0.
Using the Apache Kafka CLI: Run kafka-topics from inside a Kafka broker pod:
```
kubectl exec kafka-0 -n <namespace> -- \
  kafka-topics --bootstrap-server kafka:9071 \
    --describe --under-replicated-partitions
```
Expected: empty output (no topics listed).

If your Kafka cluster uses authentication, add --command-config <client.properties> to the command with the appropriate credentials.

Upgrade Log4j to Log4j 2

Starting in Confluent Platform 8.0, Confluent Platform only supports Log4j 2. The Log4j upgrade tasks depend on your CFK and Confluent Platform versions.

Upgrade Log4j to Log4j 2 in Confluent Platform 8.0

When you upgrade from Confluent Platform 7.x to 8.x, Confluent Platform does not automatically convert custom Log4j configurations. You must explicitly define Log4j 2 properties as described in Configure Log4j 2.

In CFK 3.x, Log4j 2 uses default properties.

Enable Log4j if using Confluent for Kubernetes 3.x to manage Confluent Platform 7.x

When upgrading CFK 2.x to 3.x to deploy and manage Confluent Platform 7.x, set the platform.confluent.io/use-log4j1=true annotation as described in Upgrade CFK.

To upgrade from Confluent Platform 7.x to 8.x, remove the annotation or set it to false during the Confluent Platform upgrade.

Basic authentication with JAAS when using CFK 3.x with Confluent Platform 7.x

CFK 3.x, by default, uses the Confluent Platform 8.x Java Authentication and Authorization Service (JAAS) class path for basic authentication. When you use CFK 3.x with Confluent Platform 7.x and have basic authentication enabled, add the platform.confluent.io/use-old-jetty9: "true" annotation to use the JAAS class path compatible with Confluent Platform 7.x.

This applies to the following components when basic authentication is enabled:

Schema Registry
Connect
ksqlDB
Control Center and Control Center (Legacy)
REST Proxy

Symptoms

Authentication attempts to component Representational State Transfer (REST) APIs return 401 Unauthorized errors.

Solution

Add the annotation to affected component custom resources (CRs) when using CFK 3.x with Confluent Platform 7.x:

apiVersion: platform.confluent.io/v1beta1
kind: SchemaRegistry
metadata:
  name: schemaregistry
  annotations:
    platform.confluent.io/use-old-jetty9: "true"

For more information, see Issue: Authentication failures when using CFK 3.0 with Confluent Platform 7.x.

When you upgrade to Confluent Platform 8.0 or later, remove this annotation.

Upgrade auto-generated certificates to configurable auto-generated certificates

Starting in CFK 2.4.0, you can use configurable certificates that CFK generates with options you specify. To upgrade certificates auto-generated in CFK 2.3.x or earlier to configurable auto-managed certificates in 2.4.x or later:

Set managedCerts.enabled: true and managedCerts.caCertificate.secretRef=<secretName> in the CFK Helm values file.
Configure other settings under managedCerts as described in Use auto-generated TLS certificates.

CFK renews certificates based on the Certificate Authority (CA) certificates and key in managedCerts.caCertificate.secretRef. CFK then rolls the corresponding Confluent clusters.

Note

CFK does not support changing the CA when moving from legacy to configurable certificates mode.

Known issues during upgrade

The following are known issues that can occur during an upgrade. Read the ones that apply to your scenario before you start, so you recognize the behavior and know the workaround.

Kafka clusters involved in Cluster Linking

When upgrading KRaft-based Kafka clusters used in Cluster Linking, upgrade brokers on the destination cluster first. Then upgrade the metadata version on the source cluster.

Connect pods restart when upgrading to CFK 2.9 and later

CFK 2.9 updates a configuration property in the Connect stateful set template. This change causes the Connect cluster to roll when you upgrade CFK to 2.9 or later.

Disabled JMX reporter when using telemetry reporter in Confluent Platform 8.0.0

In Confluent Platform 8.0.0, using Control Center or other telemetry reporters disables the Java Management Extensions (JMX) reporter. CFK uses the JMX reporter metric to validate cluster health, so the upgrade halts when the reporter is off.

Workaround: Add org.apache.kafka.common.metrics.JmxReporter to metric.reporters. Apply this override on both the Kafka and KRaftController CRs. Brokers and controllers each emit metrics through their own JMX reporter.

For example, if you previously set metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter:

kind: Kafka
spec:
  configOverrides:
    server:
      - metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter, org.apache.kafka.common.metrics.JmxReporter

kind: KRaftController
spec:
  configOverrides:
    server:
      - metric.reporters=io.confluent.telemetry.reporter.TelemetryReporter, org.apache.kafka.common.metrics.JmxReporter

Control Center crash loop with the Log4j compatibility annotation

The platform.confluent.io/use-log4j1=true annotation set in Upgrade CFK can cause the ControlCenter pod to crash loop. This happens when you run CFK 3.x with Confluent Platform 7.x and then upgrade Control Center to 2.4.x or 2.5.x. The pod logs show Control Center needs write access to the configured java.io.tmpdir (confirmed on Control Center 2.5.0).

Workaround: Remove the annotation from the ControlCenter CR only, and keep it on the other Confluent Platform 7.x components:

kubectl annotate controlcenter <CR name> \
    platform.confluent.io/use-log4j1- \
    -n <namespace>