Known Issues

The following are known issues and workarounds in Confluent for Kubernetes.

Issue: Unable to delete Kubernetes resources

Kubernetes resources can’t be deleted, for example, if the CFK pod gets deleted before other resources, if CFK can’t delete resources, or if the namespace is in termination state.

Solution: Remove the finalizer using the following command:

kubectl get <resource> --no-headers | \
  awk '{print $1 }' | \
  xargs kubectl patch <resource> -p '{"metadata":{"finalizers":[]}}' \
  --type=merge

Issue: ConfluentRoleBindings stuck in DELETING

The ConfluentRolebindings custom resources (CRs) can be stuck in the DELETING state if the associated Kafka cluster is removed.

Solution: Manually remove the finalizer for those ConfluentRolebindings CRs as shown in the following example:

Check the status of the ConfluentRolebindings custom resources:

kubectl get cfrb

The output should have the DELETING status:

NAME                         STATUS     KAFKACLUSTERID           PRINCIPAL        ROLE
c3-connect-operator-7gffem   DELETING   8itASw0_S6qDfdl72b7Uyg   User:c3          SystemAdmin
c3-ksql-operator-7gffem      DELETING   8itASw0_S6qDfdl72b7Uyg   User:c3          ResourceOwner
c3-operator-7gffem           DELETING   8itASw0_S6qDfdl72b7Uyg   User:c3          ClusterAdmin
c3-sr-operator-7gffem        DELETING   8itASw0_S6qDfdl72b7Uyg   User:c3          SystemAdmin
connect-operator-7gffem-0    DELETING   8itASw0_S6qDfdl72b7Uyg   User:connect     SystemAdmin
connect-operator-7gffem-1    DELETING   8itASw0_S6qDfdl72b7Uyg   User:connect     SystemAdmin
internal-connect-0           DELETING   8itASw0_S6qDfdl72b7Uyg   User:connect     SecurityAdmin
internal-connect-1           DELETING   8itASw0_S6qDfdl72b7Uyg   User:connect     ResourceOwner
internal-connect-2           DELETING   8itASw0_S6qDfdl72b7Uyg   User:connect     DeveloperWrite

Remove the finalizer of each ConfluentRolebindings CR:

for rb in $(kubectl get cfrb --no-headers | grep "DELETING" | awk '{print $1}'); \
  do kubectl patch cfrb $rb -p '{"metadata":{"finalizers":[]}}' \
  --type=merge; \
  done

You can remove the finalizer for all the ConfluentRolebindings CRs in a namespace, no matter their status. For example, you might remove it when the namespace is stuck in the terminating state, and you need to clean up all the ConfluentRolebindings. To remove the finalizer, run the following command:
```
for rb in $(kubectl get cfrb --no-headers -ojsonpath='{.items[*].metadata.name}'); \
  do kubectl patch cfrb $rb -p '{"metadata":{"finalizers":[]}}' \
  --type=merge; \
  done
```

Issue: KafkaTopic is stuck in DELETING/DELETE_REQUESTED state

Deleting a Kafka topic (kubectl delete kafkatopic <topic-name>) uses the Kubernetes finalizer feature to remove the topic from the destination Kafka cluster. If the finalizer fails to delete because of a network issue or unavailability of the Kafka cluster (deleted), the kubectl delete command hangs.

Solution: Patch the kafkatopic with following commands to remove the topic resource:

kubectl patch kafkatopic <topic-name> -p '{"metadata":{"finalizers":[]}}' \
  --type=merge

Delete Confluent Platform component pods

To manually delete Confluent Platform component pods, run the following command for each component.

Warning

This operation completely brings down the component. Use caution when deleting the Kafka pod because it triggers data loss.

kubectl delete pod -l platform.confluent.io/type=<cr-type>

<cr-type> is one of the following: kafka, zookeeper, schemaregistry, ksqldb, connect, controlcenter

If a pod is already in the crashloopback state, CFK doesn’t honor the changes until the pod goes back to the running state. You can use --force --grace-period=0 with the preceding command.

Block Kubernetes object reconciliation

CFK includes multiple controllers. It’s a controller’s job to ensure that, for any given object, the actual state of the world matches the desired state in the object. The state of the world includes both the cluster state, and potentially external state, like running containers for Kubelet or loadbalancers for a cloud provider. The process that ensures these states match is called reconciliation.

Solution:

To block reconciliation (often needed when upgrading), use the following command to add the annotation:
```
kubectl annotate <cr-type> <cluster_name> platform.confluent.io/block-reconcile=true
```
<cr-type> is one of the following: kafka, zookeeper, schemaregistry, ksqldb, connect, controlcenter
To unblock reconciliation, remove this annotation. If you don’t remove it, CFK ignores later changes to the CustomResource:
```
kubectl annotate <cr-type> <cluster_name> platform.confluent.io/block-reconcile-
```
To force trigger the reconciliation, add the following annotation:
```
kubectl annotate <cluster-type> <cluster_name> platform.confluent.io/force-reconcile=true
```
After the reconciliation completes, CFK automatically disables this annotation. Use this command when an auto-generated certificate expires and you need to notify CFK to create a new certificate.

Warning: Operation cannot be fulfilled, object has been modified

You can ignore the following Warning because CFK automatically reworks to apply the changes.

Operation cannot be fulfilled xxxx, the object has been modified please apply
your changes to the latest version and try again.

Solution: In most scenarios, these are benign and go away. If you continue seeing the same Warning repeatedly, create a support ticket for further investigation.

Issue: StorageClass doesn’t have the reclaimPolicy set to retain

You must configure the StorageClass (SC) of the PersistentVolume that CFK uses with reclaimPolicy: Retain. If you already created the StorageClass and its PersistentVolumes, you can’t change the StorageClass. In this case, patch the PersistentVolume as shown below.

Solution:

List the PersistentVolumes:
```
kubectl get pv
```
Check the PersistentVolume names that CFK uses and their reclaim policy in the output.
Change the PersistentVolumes that don’t have the reclaim policy set to Retain. Patch the PersistentVolumes and set their reclaim policy to Retain.
```
kubectl -n <namespace> patch pv <pv-name> \
  -p '{"spec":{"persistentVolumeReclaimPolicy": "Retain"}}'
```
Verify that the PersistentVolume has the correct reclaim policy, Retain.
```
kubectl get pv
```

Issue: CFK sends `DELETE_TOPICS` requests for auto-mirrored topics

Auto-mirrored topics are mirror topics that CFK doesn’t explicitly list in the ClusterLink CR. Kafka creates them automatically on the destination cluster when a topic on the source cluster matches the cluster link’s auto.create.mirror.topics.filters (with auto.create.mirror.topics.enable set to true).

On each reconciliation, CFK compares the mirror topics declared in the ClusterLink CR against the topics that exist on the destination cluster. Because auto-mirrored topics aren’t listed in the CR, CFK treats them as topics that shouldn’t exist and issues DELETE_TOPICS requests for them.

For auto-mirrored topics, the brokers reject these requests and log the following:

TopicDeletionDisabledException: Auto-mirrored topic cannot be deleted. The topic <topic-name> should be excluded from auto.create.mirror.topics.filters...

A related symptom occurs when you delete a topic from the source cluster. The corresponding mirror topic on the destination enters the FAILED state with the SOURCE_TOPIC_MAY_BE_DELETED failure reason, and CFK then deletes the failed topic on the next reconciliation even though it was auto-mirrored.

Resolution: Upgrade CFK to 3.2 or later, which includes the fix tracked in CFK-3888.

Issue: Scale down Kafka brokers in deployments with multiple racks or availability zones

If you notice that your replicas don’t scale down as expected, and you see the following error in the Kafka logs, this might indicate a block from the shrink workflow. You need to manually intervene to complete the scale-down process.

[ERROR] 2025-06-30 18:10:47,791 [SBK_AnomalyDetector-0]
io.confluent.databalancer.operation.EvenClusterLoadStateManager registerEvent
- Registering EvenClusterLoadEvent BALANCING_FAILED with exception
org.apache.kafka.common.errors.BalanceCannotBeImprovedException.
org.apache.kafka.common.errors.BalanceCannotBeImprovedException: SBC detected
imbalance but is not able to find reassignments to improve the balance.

CFK doesn’t support cluster shrink in setups that have multiple availability zones or racks. This section provides workaround guidance for scaling down Kafka brokers in such deployments.

For the related Kubernetes limitation, see Limitation on de-scheduler not following spread constraints.

When the Kafka brokers aren’t balanced, you need to turn off Self-Balancing, put a throttle in place, and then delete the uneven pods to make brokers balanced across nodes. Follow these steps:

Turn off the Self-Balancing feature and Kafka cluster shrinking in the Kafka CR.

Add the annotation to turn off the shrink workflow, and add the override to turn off Self-Balancing. Then apply the change with the kubectl apply -f command.
```
kind: Kafka
metadata:
  annotations:
    platform.confluent.io/enable-shrink: "false"
spec:
  replicas: 6
  configOverrides:
    server:
      - confluent.balancer.enable=false
```
Set throttles for the leader and the followers.

After Kafka brokers restart, use the following properties to set upper-bound limits on data migration during a cluster rebalance.
- leader.replication.throttled.rate
- follower.replication.throttled.rate
For details, see Limiting bandwidth usage during data migration.

Set both rates to 39321600. The default is 13107200.

These configurations can be set at the broker level or at a topic level.

To apply changes, use the kafka-configs command-line tool:
```
kafka-configs --bootstrap-server {BROKER HOST:PORT} \
  --command-config {PATH TO PROPERTIES} \
  --entity-type brokers \
  --entity-default \
  --alter \
  --add-config leader.replication.throttled.rate=39321600,follower.replication.throttled.rate=39321600
```
To check the change:
```
kafka-configs --bootstrap-server {BROKER HOST:PORT} \
  --command-config {PATH TO PROPERTIES} \
  --entity-type brokers \
  --describe
```
For more information, see the Knowledge Base article, How to manually set throttle for replication during reassignment (requires login).
With the throttling in place, begin to shrink the cluster.

Reduce the number of replicas. This should proceed without issue because you turned off the shrink workflow and Self-Balancing in the previous step.

After you’ve reduced the replica count to your desired amount, you need to manually redistribute the Kafka pods to ensure an even distribution based on your original topology constraints.

To take inventory of the current pod placements:
```
kubectl -n <namespace> get pods -o wide
```
Review the kafka-x pods and their current node and label placement. Following is an example output from the preceding command that shows a skewed distribution:
```
NAME      READY   STATUS    RESTARTS  AGE   IP          NODE           NOMINATED NODE READINESS
kafka-0   1/1     Running   0         11m   10.42.2.9   k3d-us-east-1  <none>         <none>
kafka-1   1/1     Running   0         11m   10.42.2.9   k3d-us-east-1  <none>         <none>
kafka-2   1/1     Running   0         11m   10.42.2.9   k3d-us-east-1  <none>         <none>
kafka-3   1/1     Running   0         11m   10.42.2.9   k3d-us-east-1  <none>         <none>
kafka-4   1/1     Running   0         11m   10.42.1.7   k3d-us-west-1  <none>         <none>
kafka-5   1/1     Running   0         11m   10.42.2.9   k3d-us-east-1  <none>         <none>
```
In the preceding example, the brokers are split between two regions. Five in us-east-1 and one in us-west-1 in an uneven distribution. To redistribute the pods, you can delete pods and let the Kube Scheduler schedule the pod based on the topology constraints.

Run the following command to delete a pod and let the Kube Scheduler schedule the pod based on the topology constraints. You should do this one broker at a time:
```
kubectl -n <namespace> delete pod kafka-5
```
Let the pod terminate. After Kubernetes reschedules it and it’s back up, you can move to another pod.

You should do this until you are satisfied with the distribution.

Reset the throttles.

kafka-configs --bootstrap-server {BROKER HOST:PORT} \
  --command-config {PATH TO PROPERTIES} \
  --entity-type brokers \
  --entity-default \
  --alter \
  --delete-config leader.replication.throttled.rate,follower.replication.throttled.rate

Re-enable Self-Balancing and the shrink workflow when you are satisfied with the distribution and there are no under-replicated partitions.
```
kubectl annotate kafka kafka -n <namespace> platform.confluent.io/enable-shrink-
```
```
kind: Kafka
metadata:
  annotations:
    platform.confluent.io/enable-shrink: "true"
spec:
  replicas: 6
  configOverrides:
    server:
      - confluent.balancer.enable=true
```
The Kafka pods roll once more, and CFK re-enables Self-Balancing. Self-Balancing may take a little time to read metrics and determine whether it needs to redistribute topic partitions based on the new configuration.

Issue: Kafka pods in Crashloopbackoff with the error that no space is left on device

Kafka pods go into the CrashLoopBackOff state with the following java.io.IOException. This is due to corrupted ext4 metadata on those disks.

ERROR [SharedServer id=0] Got exception while starting SharedServer
(kafka.server.SharedServer) java.io.IOException: No space left on device

Solution:

Increase the storage capacity in your Kafka CR. For details, see Expand storage.
Monitor and verify the expansion by running the commands shown below. The following commands omit namespaces for brevity.
- kubectl get pvc
  
  The CAPACITY column for all Kafka PVCs reflects the new size.
- kubectl describe pvc <pvc-name>
  
  Look for the Events section and confirm the FileSystemResizeSuccessful event.
- kubectl exec -it <kafka-pod> -- df -h /mnt/data/data0
  
  The output inside the pod (where the data volume is mounted, /mnt/data/data0) shows the new and larger total capacity.
- kubectl logs -f kafka-0 | grep DiskUsage
  
  Validate the file system exposed to the pod where the associated PersistentVolume (PV) is mounted.
  
  For example:
```
INFO DiskUsage Agent: Ping Volume{store=/mnt/data/data0 (/dev/sda),
total=xxxxx, used=xxxx, available=0, percentUsed=99.9954615035756,
percentAvailable=0.0, mountpoint='/mnt/data/data0', deviceName='/dev/sda'}
(io.confluent.agent.monitoring.DiskUsage)
```
If the StorageClass is configured with allowVolumeExpansion: true, and the underlying infrastructure supports node-side expansion, pods don’t require restarts unless there are some failures in the underlying file system.

Known Issues

Issue: Unable to delete Kubernetes resources

Issue: ConfluentRoleBindings stuck in DELETING

Issue: KafkaTopic is stuck in DELETING/DELETE_REQUESTED state

Delete Confluent Platform component pods

Block Kubernetes object reconciliation

Warning: Operation cannot be fulfilled, object has been modified

Issue: StorageClass doesn’t have the reclaimPolicy set to retain

Issue: CFK sends DELETE_TOPICS requests for auto-mirrored topics

Issue: Scale down Kafka brokers in deployments with multiple racks or availability zones

Issue: Kafka pods in Crashloopbackoff with the error that no space is left on device

Related content

Issue: CFK sends `DELETE_TOPICS` requests for auto-mirrored topics