Common Upgrade Issues

Resolve common issues encountered while upgrading CFK or Confluent Platform using the following steps.

Issue: An error returns while applying a CRD during an upgrade

As part of CFK upgrade, you need to upgrade Confluent Platform custom resource definitions (CRDs). For the difference between the CRD locations and which one to use, see Upgrade CFK.

kubectl apply -f confluent-for-kubernetes/crds/

You might get an error message similar to the following from the command:

The CustomResourceDefinition "kafkas.platform.confluent.io" is invalid:
metadata.annotations: Too long: must have at most 262144 bytes make: ***
[install-crds] Error 1

Solution: Run the following commands to apply the CRD:

kubectl apply --server-side=true --force-conflicts -f <CRD>

If running kubectl apply with the --server-side=true flag returns an error similar to the following:

Apply failed with 1 conflict: conflict with "helm" using
apiextensions.k8s.io/v1: .spec.versions Please review the fields above--they
currently have other managers.

Run kubectl apply with an additional flag, --force-conflicts:

kubectl apply --server-side=true --force-conflicts -f <CRD>

Issue: The CFK Helm release Secret exceeds the Kubernetes Secret size limit

When you run helm upgrade to upgrade CFK, you might get an error similar to the following:

Error: UPGRADE FAILED: create: failed to create: Secret
``sh.helm.release.v1.confluent-operator.v34`` is invalid: data: Too long:
may not be more than 1048576 bytes

Helm stores all the contents of the chart directory, including the crds/ directory, in a Kubernetes Secret each time you run helm upgrade. Kubernetes Secret objects have a hard limit of 1MiB. The crds/ directory in the CFK chart is large, so after Helm includes it in the Helm release Secret, the encoded content can exceed this limit and cause the upgrade to fail.

Solution: After you apply the CRDs using kubectl apply as described in Upgrade CFK, remove the crds/ directory from the chart directory before you run helm upgrade. Helm doesn’t manage the CRD lifecycle, so removing the directory doesn’t affect the CRDs already applied to the cluster.

rm -rf confluent-for-kubernetes/crds/

Then continue with the helm upgrade command as usual.

Issue: CFK fails to start up with an error about the ClusterLink CR

CFK fails to start up with the following error about ClusterLink CR:

13 W0819 02:00:08.878157       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta1.ClusterLink: json: cannot unmarshal string into Go struct field ClusterLinkStatus.items.status.mirrorTopics of        type v1beta1.MirrorTopicStatus
14 E0819 02:00:08.878189       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.ClusterLink: failed to list *v1beta1.ClusterLink: json: cannot unmarshal string into Go struct field                 ClusterLinkStatus.items.status.mirrorTopics of type v1beta1.MirrorTopicStatus

CFK 2.4.0 introduced a breaking change in the ClusterLink status. Take the following steps to fix the issue:

Remove the status from ClusterLink CRD:

kubectl patch crd clusterlinks.platform.confluent.io --type='json' \
  --patch='[{"op": "replace", "path": "/spec/versions/0/schema/openAPIV3Schema/properties/status/properties", "value": ''}]'

Confirm that the status is empty in the ClusterLink CR and that CFK comes up without errors:

kubectl -n <namespace> get clusterlink <name> -oyaml

The output should look similar to the following:

apiVersion: v1
items:
- apiVersion: platform.confluent.io/v1beta1
  kind: ClusterLink
  spec:
    destinationKafkaCluster:
      kafkaRestClassRef:
        name: destination-kafka-rest
        namespace: destination
    mirrorTopics:
    - name: demo-cl
    sourceKafkaCluster:
      bootstrapEndpoint: kafka.origin.svc.cluster.local:9071
      kafkaRestClassRef:
        name: origin-kafka-rest
        namespace: origin
  status: {}

Apply the latest CRDs again.

kubectl apply -f <CFK home>/confluent-for-kubernetes/crds

Alternatively, you can apply only the ClusterLink CRD:

kubectl apply -f <CFK home>/confluent-for-kubernetes/crds/platform.confluent.io_clusterlinks.yaml

Do a force reconcile of all the ClusterLink CRs to update the status:
```
kubectl -n <namespace> annotate clusterlink <name> \
  platform.confluent.io/force-reconcile=true
```
Alternatively, you can restart the CFK pod instead of doing the previous step, but that would reconcile all the Confluent Platform resources.

Now, the ClusterLink CR should have the correct status. For example:

mirrorTopics:
  demo-cl:
    replicationFactor: 3
    sourceTopicName: demo-cl
    status: ACTIVE
numMirrorTopics: 1
sourceKafkaClusterID: I7lBtEB6Qxq5-CaE0b202g
state: CREATED

Issue: Control Center 2.0 fails when deployed with Confluent Platform 8.0 or later

When you deploy Control Center 2.0.0 or Confluent Control Center (Legacy) with Confluent Platform 8.0 or later, you see the following error that Kafka returns.

ERROR [main] attempt=failed to create
topic=TopicInfo{name=_confluent-controlcenter-7-9-1-0-TriggerEventsStore-changelog,
partitions=12, replication=3} (io.confluent.controlcenter.KafkaHelper)
org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
config name: message.timestamp.difference.max.ms

Solution: You must use Control Center 2.2 or later with Confluent Platform 8.0 or later.

Issue: Authentication failures when using CFK 3.0 with Confluent Platform 7.x

Authentication attempts to Schema Registry, Connect, ksqlDB, Control Center, Control Center (Legacy), or REST Proxy return 401 Unauthorized errors. In Control Center, you might experience login prompt loops.

This issue occurs when you use CFK 3.0 or later with Confluent Platform 7.x.

The root cause is that CFK 3.0 and later use the Confluent Platform 8.0 JAAS class path org.eclipse.jetty.security.jaas.spi.PropertyFileLoginModule, instead of the earlier Confluent Platform 7.x class path org.eclipse.jetty.jaas.spi.PropertyFileLoginModule.

Solution

To use the JAAS class path compatible with Confluent Platform 7.x, add the annotation, platform.confluent.io/use-old-jetty9: "true", to your Confluent Platform component CR that expose REST API endpoints and have authentication enabled on those endpoints, such as Control Center, Control Center (Legacy), Schema Registry, Connect, ksqlDB, and REST Proxy.

apiVersion: platform.confluent.io/v1beta1
kind: <CP component>
metadata:
  name: controlcenter
  annotations:
    platform.confluent.io/use-old-jetty9: "true"

When you upgrade to Confluent Platform 8.0 or later, remove this annotation.