Configure and Manage KRaft for Confluent Platform Using Confluent for Kubernetes¶
In Confluent for Kubernetes (CFK), you can use ZooKeeper or KRaft for Kafka consensus and metadata storage. This document describes the configuration for KRaft in CFK.
Note
As of Confluent Platform 7.5, ZooKeeper is deprecated for new deployments. Confluent recommends KRaft mode for new deployments. For more information, see KRaft Overview.
The KRaft mode, short for Kafka Raft Metadata mode, utilizes the Raft consensus algorithm and is part of Kafka. Instead of using ZooKeeper to store cluster state and metadata, KRaft stores metadata within a Kafka metadata topic.
In CFK, you can configure, deploy, and manage KRaft with a declarative API. CFK provides a custom resource definition (CRD) for the KRaft controller (KRaftController).
KRaft in CFK works with Confluent Platform 7.4.0 and later.
- KRaft controller
- KRaft uses controllers for metadata consensus and storage. An odd number (at least 3) of KRaft controller replicas are required.
- KRaft controller listener
Communication to and among the KRaft controller nodes happens over the controller listener.
When KRaft is enabled, the controller listener must be identically configured on the controller side and broker side, including the same authentication and TLS properties.
CFK configures the controller listener on port 9074.
The high-level workflow to deploy Confluent Platform with KRaft is:
-
Alternatively, you can deploy CFK with the data recovery feature enabled. This option does not support namespace-scoped deployment because CFK needs to create the ClusterRole and ClusterRolebinding required for the feature.
Configure KRaft controllers¶
To create a KRaft controller, create and configure a KRaftController CR. The following shows the key CR settings:
kind: KRaftController
metadata:
name: --- [1]
namespace: --- [2]
spec:
replicas: --- [3]
listeners:
controller: --- [4]
authentication: --- [5]
tls:
enabled: --- [6]
externalAccess: --- [7]
type: --- [8]
configOverrides:
server: --- [9]
controllerQuorumVoters: --- [10]
[1] Required. The name of this KRaft controller.
[2] The namespace of this KRaft controller.
[3] The desired number of replicas. Must be an odd number that is 3 or higher.
A change to this setting will roll the cluster.
[4] Required. Communication to and among the KRaft controller nodes happens over this controller listener.
[5] See Configure authentication to access Kafka for configuring authentication.
[6] Set to
true
to enable TLS. See Configure Network Encryption for Confluent Platform Using Confluent for Kubernetes for configuring TLS certificates.[7] defines the external access configuration for the Kafka cluster.
[8] Required if
externalAccess
([7]) is specified. Set to the Kubernetes service for external access. Valid options areloadBalancer
,nodePort
,route
,staticForPortBasedRouting
, andstaticForHostBasedRouting
.For details on external access configuration, see Configure Networking for Confluent Platform in Confluent for Kubernetes.
[9] Required when enabling schema validation in KRaft brokers.
Check the Kafka CR in the
spec.dependencies.schemaRegistry
section, and set the same security settings in this section. For the syntax to set the settings, refer to Schema Registry security settings.[10] Required for multi-region deployment. Follow the further configuration steps in Multi-Region Deployment of Confluent Platform in Confluent for Kubernetes.
An example KRaftController CR:
kind: KRaftController
metadata:
name: kcontroller
namespace: operator
spec:
replicas: 3
listeners:
controller:
authentication:
type: plain
jaasConfig:
secretRef: kraft-secret
tls:
enabled: true
Configure Kafka for KRaft¶
For KRaft-enabled Kafka, add a kRaftController cluster reference in the
dependencies
section of the Kafka CR.
kind: Kafka
spec:
dependencies:
kRaftController: --- [1]
clusterRef: --- [2]
name: --- [3]
namespace: --- [4]
controllerListener:
authentication: --- [5]
tls: --- [6]
- [1] Kafka will fail to start if both
zookeeper
andkRaftController
are specified in thedependencies
object. - [2] A reference to a kRaftController CR. If omitted, Kafka will automatically discover a KRaft controller running in the same namespace.
- [3] The name of the kRaftController CR.
- [4] The namespace of the kRaftController CR.
- [5] Required if the controller listener on the KRaft controller cluster (kRaftController CR) has authentication configured. Must be identically configured as in the kRaftController CR. See Configure KRaft controllers.
- [6] Required if the controller listener on the KRaft controller cluster (kRaftController CR) has TLS enabled. Must be identically configured as in the kRaftController CR. See Configure KRaft controllers.
The following is an example CR of a KRaft-enabled Kafka:
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
name: kafka
namespace: operator
annotations:
platform.confluent.io/broker-id-offset: 10
spec:
dependencies:
kRaftController:
clusterRef:
name: kcontroller
namespace: operator
controllerListener:
authentication:
type: plain
jaasConfig:
secretRef: secret
tls:
enabled: true