# Known Issues The following are known issues and workarounds in Confluent for Kubernetes. ## Issue: Unable to delete Kubernetes resources Kubernetes resources can’t be deleted, for example, if the CFK pod gets deleted before other resources, if CFK can’t delete resources, or if the namespace is in termination state. **Solution:** Remove the finalizer using the following command: ```bash kubectl get --no-headers | \ awk '{print $1 }' | \ xargs kubectl patch -p '{"metadata":{"finalizers":[]}}' \ --type=merge ``` ## Issue: ConfluentRoleBindings stuck in DELETING The ConfluentRolebindings custom resources (CRs) can be stuck in the `DELETING` state if the associated Kafka cluster is removed. **Solution:** Manually remove the finalizer for those ConfluentRolebindings CRs as shown in the following example: 1. Check the status of the ConfluentRolebindings custom resources: ```bash kubectl get cfrb ``` The output should have the `DELETING` status: ```text NAME STATUS KAFKACLUSTERID PRINCIPAL ROLE c3-connect-operator-7gffem DELETING 8itASw0_S6qDfdl72b7Uyg User:c3 SystemAdmin c3-ksql-operator-7gffem DELETING 8itASw0_S6qDfdl72b7Uyg User:c3 ResourceOwner c3-operator-7gffem DELETING 8itASw0_S6qDfdl72b7Uyg User:c3 ClusterAdmin c3-sr-operator-7gffem DELETING 8itASw0_S6qDfdl72b7Uyg User:c3 SystemAdmin connect-operator-7gffem-0 DELETING 8itASw0_S6qDfdl72b7Uyg User:connect SystemAdmin connect-operator-7gffem-1 DELETING 8itASw0_S6qDfdl72b7Uyg User:connect SystemAdmin internal-connect-0 DELETING 8itASw0_S6qDfdl72b7Uyg User:connect SecurityAdmin internal-connect-1 DELETING 8itASw0_S6qDfdl72b7Uyg User:connect ResourceOwner internal-connect-2 DELETING 8itASw0_S6qDfdl72b7Uyg User:connect DeveloperWrite ``` 2. Remove the finalizer of each ConfluentRolebindings CR: ```bash for rb in $(kubectl get cfrb --no-headers | grep "DELETING" | awk '{print $1}'); \ do kubectl patch cfrb $rb -p '{"metadata":{"finalizers":[]}}' \ --type=merge; \ done ``` 3. You can remove the finalizer for all the ConfluentRolebindings CRs in a namespace, no matter their status. For example, you might remove it when the namespace is stuck in the terminating state, and you need to clean up all the ConfluentRolebindings. To remove the finalizer, run the following command: ```bash for rb in $(kubectl get cfrb --no-headers -ojsonpath='{.items[*].metadata.name}'); \ do kubectl patch cfrb $rb -p '{"metadata":{"finalizers":[]}}' \ --type=merge; \ done ``` ## Issue: KafkaTopic is stuck in DELETING/DELETE_REQUESTED state Deleting a Kafka topic (`kubectl delete kafkatopic `) uses the Kubernetes finalizer feature to remove the topic from the destination Kafka cluster. If the finalizer fails to delete because of a network issue or unavailability of the Kafka cluster (deleted), the `kubectl delete` command hangs. **Solution:** Patch the `kafkatopic` with following commands to remove the topic resource: ```bash kubectl patch kafkatopic -p '{"metadata":{"finalizers":[]}}' \ --type=merge ``` ## Delete Confluent Platform component pods To manually delete Confluent Platform component pods, run the following command for each component. #### WARNING This operation completely brings down the component. Use caution when deleting the Kafka pod because it triggers data loss. ```bash kubectl delete pod -l platform.confluent.io/type= ``` `` is one of the following: `kafka`, `zookeeper`, `schemaregistry`, `ksqldb`, `connect`, `controlcenter` If a pod is already in the `crashloopback` state, CFK doesn’t honor the changes until the pod goes back to the running state. You can use `--force --grace-period=0` with the preceding command. ## Block Kubernetes object reconciliation CFK includes multiple controllers. It’s a controller’s job to ensure that, for any given object, the actual state of the world matches the desired state in the object. The state of the world includes both the cluster state, and potentially external state, like running containers for Kubelet or loadbalancers for a cloud provider. The process that ensures these states match is called reconciliation. **Solution:** * To block reconciliation (often needed when upgrading), use the following command to add the annotation: ```bash kubectl annotate platform.confluent.io/block-reconcile=true ``` `` is one of the following: `kafka`, `zookeeper`, `schemaregistry`, `ksqldb`, `connect`, `controlcenter` * To unblock reconciliation, remove this annotation. If you don’t remove it, CFK ignores later changes to the CustomResource: ```bash kubectl annotate platform.confluent.io/block-reconcile- ``` * To force trigger the reconciliation, add the following annotation: ```bash kubectl annotate platform.confluent.io/force-reconcile=true ``` After the reconciliation completes, CFK automatically disables this annotation. Use this command when an auto-generated certificate expires and you need to notify CFK to create a new certificate. ## Warning: Operation cannot be fulfilled, object has been modified You can ignore the following `Warning` because CFK automatically reworks to apply the changes. ```text Operation cannot be fulfilled xxxx, the object has been modified please apply your changes to the latest version and try again. ``` **Solution:** In most scenarios, these are benign and go away. If you continue seeing the same `Warning` repeatedly, create a support ticket for further investigation. ## Issue: StorageClass doesn’t have the reclaimPolicy set to retain You must configure the StorageClass (SC) of the PersistentVolume that CFK uses with `reclaimPolicy: Retain`. If you already created the StorageClass and its PersistentVolumes, you can’t change the StorageClass. In this case, patch the PersistentVolume as shown below. **Solution:** 1. List the PersistentVolumes: ```bash kubectl get pv ``` Check the PersistentVolume names that CFK uses and their reclaim policy in the output. 2. Change the PersistentVolumes that don’t have the reclaim policy set to `Retain`. Patch the PersistentVolumes and set their reclaim policy to `Retain`. ```bash kubectl -n patch pv \ -p '{"spec":{"persistentVolumeReclaimPolicy": "Retain"}}' ``` 3. Verify that the PersistentVolume has the correct reclaim policy, `Retain`. ```bash kubectl get pv ``` ## Issue: CFK sends `DELETE_TOPICS` requests for auto-mirrored topics Auto-mirrored topics are mirror topics that CFK doesn’t explicitly list in the ClusterLink CR. Kafka creates them automatically on the destination cluster when a topic on the source cluster matches the cluster link’s `auto.create.mirror.topics.filters` (with `auto.create.mirror.topics.enable` set to `true`). On each reconciliation, CFK compares the mirror topics declared in the ClusterLink CR against the topics that exist on the destination cluster. Because auto-mirrored topics aren’t listed in the CR, CFK treats them as topics that shouldn’t exist and issues `DELETE_TOPICS` requests for them. For auto-mirrored topics, the brokers reject these requests and log the following: ```text TopicDeletionDisabledException: Auto-mirrored topic cannot be deleted. The topic should be excluded from auto.create.mirror.topics.filters... ``` A related symptom occurs when you delete a topic from the source cluster. The corresponding mirror topic on the destination enters the `FAILED` state with the `SOURCE_TOPIC_MAY_BE_DELETED` failure reason, and CFK then deletes the failed topic on the next reconciliation even though it was auto-mirrored. **Resolution:** Upgrade CFK to 3.2 or later, which includes the fix tracked in CFK-3888. ## Issue: Scale down Kafka brokers in deployments with multiple racks or availability zones If you notice that your replicas don’t scale down as expected, and you see the following error in the Kafka logs, this might indicate a block from the shrink workflow. You need to manually intervene to complete the scale-down process. ```text [ERROR] 2025-06-30 18:10:47,791 [SBK_AnomalyDetector-0] io.confluent.databalancer.operation.EvenClusterLoadStateManager registerEvent - Registering EvenClusterLoadEvent BALANCING_FAILED with exception org.apache.kafka.common.errors.BalanceCannotBeImprovedException. org.apache.kafka.common.errors.BalanceCannotBeImprovedException: SBC detected imbalance but is not able to find reassignments to improve the balance. ``` CFK doesn’t support cluster shrink in setups that have multiple availability zones or racks. This section provides workaround guidance for scaling down Kafka brokers in such deployments. For the related Kubernetes limitation, see [Limitation on de-scheduler not following spread constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#known-limitations). When the Kafka brokers aren’t balanced, you need to turn off Self-Balancing, put a throttle in place, and then delete the uneven pods to make brokers balanced across nodes. Follow these steps: 1. Turn off the Self-Balancing feature and Kafka cluster shrinking in the Kafka CR. Add the annotation to turn off the shrink workflow, and add the override to turn off Self-Balancing. Then apply the change with the `kubectl apply -f` command. ```yaml kind: Kafka metadata: annotations: platform.confluent.io/enable-shrink: "false" spec: replicas: 6 configOverrides: server: - confluent.balancer.enable=false ``` 2. Set throttles for the leader and the followers. After Kafka brokers restart, use the following properties to set upper-bound limits on data migration during a cluster rebalance. * `leader.replication.throttled.rate` * `follower.replication.throttled.rate` For details, see [Limiting bandwidth usage during data migration](https://docs.confluent.io/platform/current/kafka/post-deployment.html#limiting-bandwidth-usage-during-data-migration). Set both rates to `39321600`. The default is `13107200`. These configurations can be set at the broker level or at a topic level. To apply changes, use the kafka-configs command-line tool: ```bash kafka-configs --bootstrap-server {BROKER HOST:PORT} \ --command-config {PATH TO PROPERTIES} \ --entity-type brokers \ --entity-default \ --alter \ --add-config leader.replication.throttled.rate=39321600,follower.replication.throttled.rate=39321600 ``` To check the change: ```bash kafka-configs --bootstrap-server {BROKER HOST:PORT} \ --command-config {PATH TO PROPERTIES} \ --entity-type brokers \ --describe ``` For more information, see the Knowledge Base article, [How to manually set throttle for replication during reassignment](https://support.confluent.io/hc/en-us/articles/360035220412) (requires login). 3. With the throttling in place, begin to shrink the cluster. Reduce the number of replicas. This should proceed without issue because you turned off the shrink workflow and Self-Balancing in the previous step. After you’ve reduced the replica count to your desired amount, you need to manually redistribute the Kafka pods to ensure an even distribution based on your original topology constraints. To take inventory of the current pod placements: ```bash kubectl -n get pods -o wide ``` Review the `kafka-x` pods and their current node and label placement. Following is an example output from the preceding command that shows a skewed distribution: ```text NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS kafka-0 1/1 Running 0 11m 10.42.2.9 k3d-us-east-1 kafka-1 1/1 Running 0 11m 10.42.2.9 k3d-us-east-1 kafka-2 1/1 Running 0 11m 10.42.2.9 k3d-us-east-1 kafka-3 1/1 Running 0 11m 10.42.2.9 k3d-us-east-1 kafka-4 1/1 Running 0 11m 10.42.1.7 k3d-us-west-1 kafka-5 1/1 Running 0 11m 10.42.2.9 k3d-us-east-1 ``` In the preceding example, the brokers are split between two regions. Five in `us-east-1` and one in `us-west-1` in an uneven distribution. To redistribute the pods, you can delete pods and let the Kube Scheduler schedule the pod based on the topology constraints. Run the following command to delete a pod and let the Kube Scheduler schedule the pod based on the topology constraints. You should do this one broker at a time: ```bash kubectl -n delete pod kafka-5 ``` Let the pod terminate. After Kubernetes reschedules it and it’s back up, you can move to another pod. You should do this until you are satisfied with the distribution. 4. Reset the throttles. ```bash kafka-configs --bootstrap-server {BROKER HOST:PORT} \ --command-config {PATH TO PROPERTIES} \ --entity-type brokers \ --entity-default \ --alter \ --delete-config leader.replication.throttled.rate,follower.replication.throttled.rate ``` 5. Re-enable Self-Balancing and the shrink workflow when you are satisfied with the distribution and there are no under-replicated partitions. ```bash kubectl annotate kafka kafka -n platform.confluent.io/enable-shrink- ``` ```yaml kind: Kafka metadata: annotations: platform.confluent.io/enable-shrink: "true" spec: replicas: 6 configOverrides: server: - confluent.balancer.enable=true ``` The Kafka pods roll once more, and CFK re-enables Self-Balancing. Self-Balancing may take a little time to read metrics and determine whether it needs to redistribute topic partitions based on the new configuration. ## Issue: Kafka pods in Crashloopbackoff with the error that no space is left on device Kafka pods go into the CrashLoopBackOff state with the following `java.io.IOException`. This is due to corrupted ext4 metadata on those disks. ```text ERROR [SharedServer id=0] Got exception while starting SharedServer (kafka.server.SharedServer) java.io.IOException: No space left on device ``` **Solution:** 1. Increase the storage capacity in your Kafka CR. For details, see [Expand storage](co-scale-storage.md#co-expand-storage). 2. Monitor and verify the expansion by running the commands shown below. The following commands omit namespaces for brevity. * `kubectl get pvc` The `CAPACITY` column for all Kafka PVCs reflects the new size. * `kubectl describe pvc ` Look for the `Events` section and confirm the `FileSystemResizeSuccessful` event. * `kubectl exec -it -- df -h /mnt/data/data0` The output inside the pod (where the data volume is mounted, `/mnt/data/data0`) shows the new and larger total capacity. * `kubectl logs -f kafka-0 | grep DiskUsage` Validate the file system exposed to the pod where the associated PersistentVolume (PV) is mounted. For example: ```text INFO DiskUsage Agent: Ping Volume{store=/mnt/data/data0 (/dev/sda), total=xxxxx, used=xxxx, available=0, percentUsed=99.9954615035756, percentAvailable=0.0, mountpoint='/mnt/data/data0', deviceName='/dev/sda'} (io.confluent.agent.monitoring.DiskUsage) ``` If the `StorageClass` is configured with `allowVolumeExpansion: true`, and the underlying infrastructure supports node-side expansion, pods don’t require restarts unless there are some failures in the underlying file system. ## Related content For additional details about troubleshooting CFK issues, see: Confluent Support (requires login): [How to set SSL debug logging for Confluent Platform component deployed with Confluent for Kubernetes](https://support.confluent.io/hc/en-us/articles/14600936528276).