Kafka Raft (KRaft) in Confluent Platform Preview¶
Important
Kafka Raft (KRaft) metadata mode is currently in preview and is intended for evaluation use in development and testing environments only, and not for production use. You should try out KRaft mode on a new, separate, cluster.
Currently, when you run Confluent Platform, you must also run ZooKeeper. The Zookeeper service stores persistent cluster metadata and has been a critical component of Apache Kafka® and Confluent Platform up to this point.
Now, Kafka is in the process of moving from storing metadata in ZooKeeper, to storing metadata directly in Kafka. With Confluent Platform 7.0.0, Kafka Raft (KRaft) metadata mode is in preview so that you can try it out.
The new KRaft mode provides a quorum controller service that uses an event-based variant of the Raft consensus protocol. This simplifies Kafka’s architecture by consolidating responsibility for metadata into Kafka itself, rather than splitting it between ZooKeeper and Kafka.
The following image provides a simple illustration of the difference between running with ZooKeeper and KRaft for managing metadata.
To learn more about KRaft mode and ZooKeeper-less Kafka, see A First Glimpse of a Kafka without Zookeeper.
Prepare for KRaft GA¶
Important
Confluent Cloud customers do not need to change their client configurations and are not impacted by Confluent Platform moving to KRaft to manage metadata.
In the coming year, KRaft mode will become the default metadata mode of Confluent Platform. It’s important that you start preparing for this change. The Confluent Platform 7.0 release includes all the API breaking changes caused by the move to KRaft and should help provide a seamless transition in the future when the move to KRaft is completed.
You should spend some time trying out Kafka in KRaft mode in a local, non-production, setting. It is also important to investigate changing your clients’ and tools’ source for metadata from ZooKeeper over to the brokers directly, as described in Preparing Your Clients and Tools for KIP-500: ZooKeeper Removal from Apache Kafka. The following table outlines how accessing some services and configuration values will change.
| Feature | With ZooKeeper | With KRaft (without ZooKeeper) |
|---|---|---|
| Clients and services | zookeeper.connect=zookeeper:2181 |
bootstrap.servers=broker:9092 |
| Schema Registry | kafkastore.connection.url=zookeeper:2181 |
kafkastore.bootstrap.servers=broker:9092 |
| Administrative tools | kafka-topics --zookeeper zookeeper:2181 |
|
| Retrieve Kafka cluster ID | zookeeper-shell zookeeper:2181 get/cluster/id |
From the command line, use kafka-metadata-quorum
or confluent cluster describe --url,
or view metadata.properties
or http://broker:8090 --output json |
Current limitations and known issues¶
- The Multi-Region Clusters Confluent Platform features require ZooKeeper, and so does not currently work with KRaft.
- Cluster Linking for Confluent Platform between a source cluster running Confluent Platform 7.0.x or earlier (non-KRaft) and a destination cluster running in KRaft mode is not supported. Link creation may succeed, but the connection will ultimately fail. To work around this issue, make sure the source cluster is running Confluent Platform version 7.1.0 or later. If you have links from a Confluent Platform source cluster to a Confluent Cloud destination cluster, you must upgrade your source clusters to Confluent Platform 7.1.0 or later to avoid this issue.
Stay tuned¶
Watch the Confluent blog and Confluent Platform 7.3 Release Notes for additional announcements about the Confluent Platform adoption of KRaft.