<a id="co-troubleshoot-upgrade"></a>

# Common Upgrade Issues

Resolve common issues encountered while upgrading CFK or Confluent Platform using the
following steps.

## Issue: An error returns while applying a CRD during an upgrade

As part of CFK upgrade, you need to upgrade Confluent Platform custom resource definitions
(CRDs). For the difference between the CRD locations and which one to use,
see [Upgrade CFK](co-upgrade-cfk.md#co-upgrade-cfk).

```bash
kubectl apply -f confluent-for-kubernetes/crds/
```

You might get an error message similar to the following from the command:

```text
The CustomResourceDefinition "kafkas.platform.confluent.io" is invalid:
metadata.annotations: Too long: must have at most 262144 bytes make: ***
[install-crds] Error 1
```

**Solution:** Run the following commands to apply the CRD:

```bash
kubectl apply --server-side=true --force-conflicts -f <CRD>
```

If running `kubectl apply` with the `--server-side=true` flag returns an
error similar to the following:

```text
Apply failed with 1 conflict: conflict with "helm" using
apiextensions.k8s.io/v1: .spec.versions Please review the fields above--they
currently have other managers.
```

Run `kubectl apply` with an additional flag, `--force-conflicts`:

```bash
kubectl apply --server-side=true --force-conflicts -f <CRD>
```

## Issue: The CFK Helm release Secret exceeds the Kubernetes Secret size limit

When you run `helm upgrade` to upgrade CFK, you might get an error similar
to the following:

```text
Error: UPGRADE FAILED: create: failed to create: Secret
``sh.helm.release.v1.confluent-operator.v34`` is invalid: data: Too long:
may not be more than 1048576 bytes
```

Helm stores all the contents of the chart directory, including the
`crds/` directory, in a Kubernetes `Secret` each time you run `helm
upgrade`. Kubernetes `Secret` objects have a hard limit of 1MiB. The
`crds/` directory in the CFK chart is large, so after Helm includes it
in the Helm release `Secret`, the encoded content can exceed this limit
and cause the upgrade to fail.

**Solution:** After you apply the CRDs using `kubectl apply` as described
in [Upgrade CFK](co-upgrade-cfk.md#co-upgrade-cfk), remove the `crds/` directory from the chart
directory before you run `helm upgrade`. Helm doesn’t manage the CRD
lifecycle, so removing the directory doesn’t affect the CRDs already applied
to the cluster.

```bash
rm -rf confluent-for-kubernetes/crds/
```

Then continue with the `helm upgrade` command as usual.

## Issue: CFK fails to start up with an error about the ClusterLink CR

CFK fails to start up with the following error about ClusterLink CR:

```text
13 W0819 02:00:08.878157       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta1.ClusterLink: json: cannot unmarshal string into Go struct field ClusterLinkStatus.items.status.mirrorTopics of        type v1beta1.MirrorTopicStatus
14 E0819 02:00:08.878189       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.ClusterLink: failed to list *v1beta1.ClusterLink: json: cannot unmarshal string into Go struct field                 ClusterLinkStatus.items.status.mirrorTopics of type v1beta1.MirrorTopicStatus
```

CFK 2.4.0 introduced a breaking change in the ClusterLink status. Take
the following steps to fix the issue:

1. Remove the status from ClusterLink CRD:
   ```bash
   kubectl patch crd clusterlinks.platform.confluent.io --type='json' \
     --patch='[{"op": "replace", "path": "/spec/versions/0/schema/openAPIV3Schema/properties/status/properties", "value": ''}]'
   ```
2. Confirm that the status is empty in the ClusterLink CR and that CFK comes up
   without errors:
   ```bash
   kubectl -n <namespace> get clusterlink <name> -oyaml
   ```

   The output should look similar to the following:
   ```yaml
   apiVersion: v1
   items:
   - apiVersion: platform.confluent.io/v1beta1
     kind: ClusterLink
     spec:
       destinationKafkaCluster:
         kafkaRestClassRef:
           name: destination-kafka-rest
           namespace: destination
       mirrorTopics:
       - name: demo-cl
       sourceKafkaCluster:
         bootstrapEndpoint: kafka.origin.svc.cluster.local:9071
         kafkaRestClassRef:
           name: origin-kafka-rest
           namespace: origin
     status: {}
   ```
3. Apply the latest CRDs again.
   ```bash
   kubectl apply -f <CFK home>/confluent-for-kubernetes/crds
   ```

   Alternatively, you can apply only the ClusterLink CRD:
   ```bash
   kubectl apply -f <CFK home>/confluent-for-kubernetes/crds/platform.confluent.io_clusterlinks.yaml
   ```
4. Do a force reconcile of all the ClusterLink CRs to update the status:
   ```bash
   kubectl -n <namespace> annotate clusterlink <name> \
     platform.confluent.io/force-reconcile=true
   ```

   Alternatively, you can restart the CFK pod instead of doing the previous
   step, but that would reconcile all the Confluent Platform resources.
5. Now, the ClusterLink CR should have the correct status. For example:
   ```bash
   mirrorTopics:
     demo-cl:
       replicationFactor: 3
       sourceTopicName: demo-cl
       status: ACTIVE
   numMirrorTopics: 1
   sourceKafkaClusterID: I7lBtEB6Qxq5-CaE0b202g
   state: CREATED
   ```

## Issue: Control Center 2.0 fails when deployed with Confluent Platform 8.0 or later

When you deploy Control Center 2.0.0 or Confluent Control Center (Legacy) with Confluent Platform 8.0 or later, you see the
following error that Kafka returns.

```text
ERROR [main] attempt=failed to create
topic=TopicInfo{name=_confluent-controlcenter-7-9-1-0-TriggerEventsStore-changelog,
partitions=12, replication=3} (io.confluent.controlcenter.KafkaHelper)
org.apache.kafka.common.errors.InvalidConfigurationException: Unknown topic
config name: message.timestamp.difference.max.ms
```

**Solution:** You must use Control Center 2.2 or later with Confluent Platform 8.0 or later.

<a id="co-jaas-class-change"></a>

## Issue: Authentication failures when using CFK 3.0 with Confluent Platform 7.x

Authentication attempts to Schema Registry, Connect, ksqlDB, Control Center, Control Center (Legacy),
or REST Proxy return `401 Unauthorized` errors. In Control Center, you might
experience login prompt loops.

This issue occurs when you use CFK 3.0 or later with Confluent Platform 7.x.

The root cause is that CFK 3.0 and later use the Confluent Platform 8.0 JAAS class path
`org.eclipse.jetty.security.jaas.spi.PropertyFileLoginModule`, instead of
the earlier Confluent Platform 7.x class path
`org.eclipse.jetty.jaas.spi.PropertyFileLoginModule`.

**Solution**

To use the JAAS class path compatible with Confluent Platform 7.x, add the
annotation, `platform.confluent.io/use-old-jetty9: "true"`, to your Confluent Platform
component CR that expose REST API endpoints and have authentication enabled on
those endpoints, such as Control Center, Control Center (Legacy), Schema Registry, Connect, ksqlDB, and
REST Proxy.

```yaml
apiVersion: platform.confluent.io/v1beta1
kind: <CP component>
metadata:
  name: controlcenter
  annotations:
    platform.confluent.io/use-old-jetty9: "true"
```

When you upgrade to Confluent Platform 8.0 or later, remove this annotation.