Configure Kafka Rack Awareness for Confluent Platform Using Confluent for Kubernetes Blueprints

This topic describes how to configure rack awareness in Confluent for Kubernetes (CFK) Blueprints.

To minimize data loss in the event of a rack failure, the Rack Awareness feature in Kafka spreads replicas of the same partition across different racks.

When you enable rack awareness in CFK Blueprints, Kubernetes Availability Zones (AZs) are treated as racks. CFK Blueprints maps each Kafka rack to an AZ and places Kafka partition replicas across AZs.

For more information about Kafka rack awareness, see Balancing Replicas Across Racks.

Enable rack awareness

1. Check the Kubernetes node labels.

   The Kubernetes cluster must have node labels set for the fault domain, and the nodes in each zone should have the same node label.

   Use the following command to check the node labels for topology.kubernetes.io/zone:

   kubectl get node \
     -o=custom-columns=NODE:.metadata.name,ZONE:.metadata.labels."topology\.kubernetes\.io/zone" \
     | sort -k2
   

2. Enable rack awareness.

   There are two ways to configure rack awareness.

   • From the KafkaClusterClass. With this model, every cluster created from this class (part of the Blueprint) will have rack awareness capability enabled.
   • From the KafkaCluster. With this model, it will be per KafkaCluster. This is recommended model if every cluster has a different label requirement.

   If both places are configured, KafkaCluster takes precedence.

   • Enable rack awareness in the KafkfClusterClass CR:

     kind: KafkaClusterClass
     spec:
       provisioner:
         cfk:
           rackAssignment:
             nodeLabels:
               - topology.kubernetes.io/zone
     

   • Enable rack awareness in the KafkaCluster CR:

     kind: KafkaCluster
     spec:
       rackAssignment:
         nodeLabels:
           - topology.kubernetes.io/zone