Configure KRaft

Use Kafka Raft (KRaft) for Apache Kafka® consensus and metadata storage with Confluent for Kubernetes (CFK). This topic covers the KRaft configuration options CFK supports.

KRaft utilizes the Raft consensus algorithm and is part of Kafka. KRaft stores metadata within a Kafka metadata topic. For detailed information, see KRaft Overview.

KRaft in CFK works with Confluent Platform 7.4.0 and later.

For Confluent Platform 7.9 and later, you can also configure a dynamic quorum. See Configure Dynamic KRaft Quorum.

In CFK, you can configure, deploy, and manage KRaft with a declarative API. CFK provides a custom resource definition (CRD) for the KRaft controller (KRaftController), and the custom resource (CR) configures the following:

KRaft controller

KRaft uses controllers for metadata consensus and storage. An odd number (at least 3) of KRaft controller replicas are required.

KRaft controller listener

Communication to and among the KRaft controller nodes happens over the controller listener.

When KRaft is enabled, the controller listener must be identically configured on the controller side and broker side, including the same authentication and TLS properties.

CFK configures the controller listener on port 9074.

The high-level workflow to deploy Confluent Platform with KRaft is:

Deploy CFK.

Alternatively, you can deploy CFK with the data recovery feature enabled. This option does not support namespace-scoped deployment because CFK needs to create the ClusterRole and ClusterRoleBinding required for the feature.
Create KRaft controllers.
Add a reference to the KRaft controller in the Kafka CR.

Configure KRaft controllers

To create a KRaft controller, create and configure a KRaftController CR. The following shows the key CR settings:

kind: KRaftController
metadata:
  name:                                    --- [1]
  namespace:                               --- [2]
spec:
  replicas:                                --- [3]
  listeners:
    controller:                            --- [4]
      authentication:                      --- [5]
      tls:
        enabled:                           --- [6]
      externalAccess:                      --- [7]
        type:                              --- [8]
  controllerQuorumVoters:                  --- [9]
  configOverrides:                         --- [10]
    server:
      - default.replication.factor=        --- [11]
      - listener                           --- [12]
  dependencies:
    schemaRegistry:                        --- [13]

[1] Required. The name of this KRaft controller.
[2] The namespace of this KRaft controller.
[3] The desired number of replicas. Must be an odd number that is 3 or higher.

A change to this setting rolls the cluster.
[4] Required. Communication to and among the KRaft controller nodes happens over this controller listener.
[5] See Authentication for Kafka and KRaft for configuring authentication.
[6] Set to true to enable Transport Layer Security (TLS). See Network encryption for configuring TLS certificates.
[7] Defines the external access configuration for the KRaft controller nodes.
[8] Required if externalAccess ([7]) is specified. Set to a Kubernetes service type for external access. Valid options are loadBalancer, nodePort, route, staticForPortBasedRouting, and staticForHostBasedRouting.

For details on external access configuration, see Configure Networking.
[9] Required for multi-region deployment. Follow the further configuration steps in Set up Multi-Region Clusters (MRC) with KRaft.
[10] Required. Use the configOverrides to set the matching properties as set in the Kafka CR. Property [11] is always required; [12] is required only when authentication is enabled for replication listeners.
[11] Required. The default replication factor is the number of Kafka replicas. The KRaft controller needs this parameter to interact with the Kafka brokers for some features, such as Self-Balancing Clusters and Metrics Reporter. You must explicitly set it for KRaft to match the number of Kafka replicas (spec.replicas in the Kafka CR).

Use configOverrides to set the property because the KRaft CR does not directly support the property. For example:
```
spec:
  configOverrides:
    server:
      - default.replication.factor=3
```

[12] Required when authentication is enabled for Kafka replication listeners. Set the same properties set in the Kafka CR, under spec.listeners.replication.authentication.

The following are sample configurations for mutual TLS (mTLS) authentication among replication listeners:

spec:
  configOverrides:
    server:
      - listener.name.replication.ssl.client.auth=required
      - listener.name.replication.ssl.key.password=${file:/vault/secrets/kafka-tls/jksPassword.txt:jksPassword}
      - listener.name.replication.ssl.keystore.location=/vault/secrets/kafka-tls/keystore.jks
      - listener.name.replication.ssl.keystore.password=${file:/vault/secrets/kafka-tls/jksPassword.txt:jksPassword}
      - listener.name.replication.ssl.principal.mapping.rules=RULE:.*CN[\s]?=[\s]?([a-zA-Z0-9._]*)?.*/$1/
      - listener.name.replication.ssl.truststore.location=/vault/secrets/kafka-tls/truststore.jks
      - listener.name.replication.ssl.truststore.password=${file:/vault/secrets/kafka-tls/jksPassword.txt:jksPassword}
      - listener.security.protocol.map=CONTROLLER:SSL,REPLICATION:SSL

[13] Required when the Kafka CR has a dependency on Schema Registry in the spec.dependencies.schemaRegistry section. Set the same Schema Registry dependency settings you set in the Kafka CR here.

An example KRaftController CR:

kind: KRaftController
metadata:
  name: kcontroller
  namespace: operator
spec:
  replicas: 3
  listeners:
    controller:
      authentication:
        type: plain
        jaasConfig:
          secretRef: kraft-secret
      tls:
        enabled: true
  dependencies:
    schemaRegistry:
      authentication:
        basic:
          secretRef: kafka-sr-credential
        type: basic
      tls:
        enabled: true
  configOverrides:
    server:
      - default.replication.factor=3

Configure Kafka for KRaft

For KRaft-enabled Kafka, add a kRaftController cluster reference in the dependencies section of the Kafka CR.

kind: Kafka
spec:
  dependencies:
    kRaftController:      --- [1]
      clusterRef:         --- [2]
        name:             --- [3]
        namespace:        --- [4]
      controllerListener:
        authentication:   --- [5]
        tls:              --- [6]

[1] Kafka fails to start if both zookeeper and kRaftController are specified in the dependencies object.
[2] A reference to a kRaftController CR. If omitted, Kafka automatically discovers a KRaft controller running in the same namespace.
[3] The name of the kRaftController CR.
[4] The namespace of the kRaftController CR.
[5] Required if the controller listener on the KRaft controller cluster (kRaftController CR) has authentication configured. Must be identically configured as in the kRaftController CR. See Configure KRaft controllers.
[6] Required if the controller listener on the KRaft controller cluster (kRaftController CR) has TLS enabled. Must be identically configured as in the kRaftController CR. See Configure KRaft controllers.

The following is an example CR of a KRaft-enabled Kafka:

apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: operator
  annotations:
    platform.confluent.io/broker-id-offset: 10
spec:
  dependencies:
    kRaftController:
      clusterRef:
        name: kcontroller
        namespace: operator

      controllerListener:
        authentication:
          type: plain
          jaasConfig:
            secretRef: secret
        tls:
          enabled: true