Quick Command Reference for ZooKeeper to KRaft Migration

This topic provides quick access to frequently used commands during migration. Commands are organized by migration phase for easy reference.

Tip

Keep this page open in a separate browser tab for quick reference during migration.

Verify pre-migration requirements

# Check Confluent for Kubernetes (CFK) version
kubectl get deployment confluent-operator -n <namespace> \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

# Check Confluent Platform version
kubectl get kafka <kafka-name> -n <namespace> \
  -o jsonpath='{.spec.image.application}'

# Check current inter-broker protocol version
kubectl exec <kafka-pod-name> -n <namespace> -- \
  grep "inter.broker.protocol.version" \
  /opt/confluentinc/etc/kafka/kafka.properties

# Verify Kafka pods are running
kubectl get pods -n <namespace> -l app=kafka

# Verify ZooKeeper is healthy
kubectl get pods -n <namespace> -l app=zookeeper

# Check for existing KRaftController
kubectl get kraftcontroller -n <namespace>
# Expected: No resources found

# If RBAC is enabled, verify MDS endpoint
kubectl exec <kafka-pod-name> -n <namespace> -- curl -k -s -o /dev/null \
  -w "HTTP %{http_code}\n" \
  https://<kafka-bootstrap-server>:8090/security/1.0/authenticate
# Expected: HTTP 401

Monitor migration progress

# Check migration phase
kubectl get kraftmigrationjob <migration-job-name> -n <namespace> \
  -o jsonpath='{.status.phase}'

# Check migration subphase
kubectl get kraftmigrationjob <migration-job-name> -n <namespace> \
  -o jsonpath='{.status.subPhase}'

# Watch migration progress in real time
watch -n 5 'kubectl get kraftmigrationjob <migration-job-name> -n <namespace> \
  -o jsonpath="{.status.phase} - {.status.subPhase}"'

# Check CFK Operator logs for migration status
kubectl logs deployment/confluent-operator -n <namespace> --tail=20 | \
  grep "Migration current status"

# Monitor pod status during rolls
watch kubectl get pods -n <namespace>

Validate DUAL-WRITE phase

# Verify DUAL-WRITE mode (Method 1 - most reliable)
kubectl get kraftmigrationjob <migration-job-name> -n <namespace> \
  -o jsonpath='{.status.phase}'
# Expected: DUAL-WRITE

# Verify DUAL-WRITE mode (Method 2 - via Jolokia)
kubectl exec <kafka-pod-name> -n <namespace> -- curl -s \
  http://localhost:7777/jolokia/read/kafka.controller:type=KafkaController,name=ZkMigrationState \
  | jq '.value.Value'
# Expected: 1 (DUAL_WRITE state)

# Check Kafka broker status
kubectl get kafka <kafka-name> -n <namespace> -o jsonpath='{.status.phase}'
# Expected: RUNNING

# Check KRaftController status
kubectl get kraftcontroller <kraftcontroller-name> -n <namespace> \
  -o jsonpath='{.status.phase}'
# Expected: RUNNING

Finalization commands

# Finalize migration (point of no return)
kubectl annotate kraftmigrationjob <migration-job-name> \
  platform.confluent.io/kraft-migration-trigger-finalize-to-kraft=true \
  -n <namespace>

# Monitor finalization progress
kubectl get kraftmigrationjob <migration-job-name> -n <namespace> -w

Post-migration tasks

# Release migration locks
kubectl annotate kraftmigrationjob <migration-job-name> \
  platform.confluent.io/kraft-migration-release-cr-lock=true \
  -n <namespace>

# Verify locks are released
kubectl get kafka <kafka-name> -n <namespace> -o yaml | \
  grep kraft-migration-cr-lock
# Should return nothing

# Download updated Kafka CR
kubectl get kafka <kafka-name> -n <namespace> -o yaml > kafka-kraft-mode.yaml

# Download KRaftController CR
kubectl get kraftcontroller <kraftcontroller-name> -n <namespace> -o yaml > kraftcontroller.yaml

# Verify Kafka uses KRaftController (not ZooKeeper)
kubectl get kafka <kafka-name> -n <namespace> \
  -o jsonpath='{.spec.dependencies.kRaftController.clusterRef.name}'

Troubleshooting commands

# Check for errors in CFK Operator logs
kubectl logs deployment/confluent-operator -n <namespace> --tail=50 | \
  grep -i error

# Check Kafka pod logs for errors
kubectl logs <kafka-pod-name> -n <namespace> --tail=50 | \
  grep -E "ERROR|FATAL"

# Check KRaftController logs for errors
kubectl logs <kraftcontroller-pod-name> -n <namespace> --tail=50 | \
  grep -E "ERROR|FATAL"

# Get full migration job status
kubectl get kraftmigrationjob <migration-job-name> -n <namespace> -o yaml

# Check pod events
kubectl describe pod <pod-name> -n <namespace>

Rollback commands

Warning

Rollback is only possible during DUAL-WRITE phase. For complete instructions, see Rollback to ZooKeeper.

# Trigger rollback to ZooKeeper
kubectl annotate kraftmigrationjob <migration-job-name> \
  platform.confluent.io/kraft-migration-trigger-rollback-to-zk=true \
  --overwrite -n <namespace>

# Delete controller node from ZooKeeper (exec into Kafka pod first)
kubectl exec -it <kafka-pod-name> -n <namespace> -- bash
zookeeper-shell <zkhost:zkport> \
  -zk-tls-config-file /opt/confluentinc/etc/kafka/zookeeper-client.properties \
  deleteall /<kafka-cr-name>-<kafka-cr-namespace>/controller

# Delete migration node from ZooKeeper
zookeeper-shell <zkhost:zkport> \
  -zk-tls-config-file /opt/confluentinc/etc/kafka/zookeeper-client.properties \
  deleteall /<kafka-cr-name>-<kafka-cr-namespace>/migration

# Continue rollback after node removal
kubectl annotate kraftmigrationjob <migration-job-name> \
  platform.confluent.io/continue-kraft-migration-post-zk-node-removal=true \
  --overwrite -n <namespace>