Configure and Manage KRaft for Confluent Platform Using Confluent for Kubernetes

In Confluent for Kubernetes (CFK), you can use ZooKeeper or Kafka Raft for Kafka consensus and metadata storage. This document describes the configuration for KRaft in CFK.

Note

As of Confluent Platform 7.5, ZooKeeper is deprecated for new deployments. Confluent recommends KRaft mode for new deployments. For more information, see KRaft Overview.

KRaft, short for Kafka Raft Metadata, utilizes the Raft consensus algorithm and is part of Kafka. Instead of using ZooKeeper to store cluster state and metadata, KRaft stores metadata within a Kafka metadata topic.

KRaft in CFK works with Confluent Platform 7.4.0 and later.

In CFK, you can configure, deploy, and manage KRaft with a declarative API. CFK provides a custom resource definition (CRD) for the KRaft controller (KRaftController), and the CR configures the following:

KRaft controller
KRaft uses controllers for metadata consensus and storage. An odd number (at least 3) of KRaft controller replicas are required.
KRaft controller listener

Communication to and among the KRaft controller nodes happens over the controller listener.

When KRaft is enabled, the controller listener must be identically configured on the controller side and broker side, including the same authentication and TLS properties.

CFK configures the controller listener on port 9074.

The high-level workflow to deploy Confluent Platform with KRaft is:

  1. Deploy CFK.

    Alternatively, you can deploy CFK with the data recovery feature enabled. This option does not support namespace-scoped deployment because CFK needs to create the ClusterRole and ClusterRolebinding required for the feature.

  2. Create KRaft controllers.

  3. Add a reference to the KRaft controller in the Kafka CR.

Configure KRaft controllers

To create a KRaft controller, create and configure a KRaftController CR. The following shows the key CR settings:

kind: KRaftController
metadata:
  name:                                    --- [1]
  namespace:                               --- [2]
spec:
  replicas:                                --- [3]
  listeners:
    controller:                            --- [4]
      authentication:                      --- [5]
      tls:
        enabled:                           --- [6]
      externalAccess:                      --- [7]
        type:                              --- [8]
  configOverrides:
    server:                                --- [9]
  controllerQuorumVoters:                  --- [10]

  • [1] Required. The name of this KRaft controller.

  • [2] The namespace of this KRaft controller.

  • [3] The desired number of replicas. Must be an odd number that is 3 or higher.

    A change to this setting will roll the cluster.

  • [4] Required. Communication to and among the KRaft controller nodes happens over this controller listener.

  • [5] See Configure authentication to access Kafka for configuring authentication.

  • [6] Set to true to enable TLS. See Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes for configuring TLS certificates.

  • [7] defines the external access configuration for the Kafka cluster.

  • [8] Required if externalAccess ([7]) is specified. Set to the Kubernetes service for external access. Valid options are loadBalancer, nodePort, route, staticForPortBasedRouting, and staticForHostBasedRouting.

    For details on external access configuration, see Configure Networking for Confluent Platform in Confluent for Kubernetes.

  • [9] Required when enabling schema validation in KRaft brokers.

    Check the Kafka CR in the spec.dependencies.schemaRegistry section, and set the same security settings in this section. For the syntax to set the settings, refer to Schema Registry security settings.

  • [10] Required for multi-region deployment. Follow the further configuration steps in Configure MRC with KRaft.

An example KRaftController CR:

kind: KRaftController
metadata:
  name: kcontroller
  namespace: operator
spec:
  replicas: 3
  listeners:
    controller:
      authentication:
        type: plain
        jaasConfig:
          secretRef: kraft-secret
      tls:
        enabled: true

Configure Kafka for KRaft

For KRaft-enabled Kafka, add a kRaftController cluster reference in the dependencies section of the Kafka CR.

kind: Kafka
spec:
  dependencies:
    kRaftController:      --- [1]
      clusterRef:         --- [2]
        name:             --- [3]
        namespace:        --- [4]
      controllerListener:
        authentication:   --- [5]
        tls:              --- [6]
  • [1] Kafka will fail to start if both zookeeper and kRaftController are specified in the dependencies object.
  • [2] A reference to a kRaftController CR. If omitted, Kafka will automatically discover a KRaft controller running in the same namespace.
  • [3] The name of the kRaftController CR.
  • [4] The namespace of the kRaftController CR.
  • [5] Required if the controller listener on the KRaft controller cluster (kRaftController CR) has authentication configured. Must be identically configured as in the kRaftController CR. See Configure KRaft controllers.
  • [6] Required if the controller listener on the KRaft controller cluster (kRaftController CR) has TLS enabled. Must be identically configured as in the kRaftController CR. See Configure KRaft controllers.

The following is an example CR of a KRaft-enabled Kafka:

apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: operator
  annotations:
    platform.confluent.io/broker-id-offset: 10
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kcontroller
        namespace: operator

      controllerListener:
        authentication:
          type: plain
          jaasConfig:
            secretRef: secret
        tls:
          enabled: true