Schema Registry Single Datacenter Setup

Within a single datacenter or location, a multi-node, multi-broker cluster provides Kafka data replication across the nodes.

Producers write and consumers read data to/from topic partition leaders. Leaders replicate data to followers so that messages are copied to more than one broker.

You can configure parameters on producers and consumers to optimize your single cluster deployment for various goals, including message durability and high availability.

Kafka producers can set the acks configuration parameter to control when a write is considered successful. For example, setting producers to acks=all requires other brokers in the cluster acknowledge receiving the data before the leader broker responds to the producer.

If a leader broker fails, the Kafka cluster recovers when a follower broker is elected leader and client applications can continue to write and read messages through the new leader.

Kafka Election

Important Settings

kafkastore.bootstrap.servers This should point to the primary Kafka cluster (DC A in this example). is used as the consumer For single datacenter setup, make this setting the same for all nodes in the cluster. When set, overrides for the Kafka group when Kafka is used for primary election. (Without this configuration, will be “schema-registry”.)

master.eligibility In a single datacenter setup, all Schema Registry instances will be local to the Kafka cluster and should have master.eligibility set to true.

Run Book

Let’s say you have Schema Registry running in a single datacenter , and the master node goes down; what do you do? First, note that the remaining Schema Registry instances can continue to serve requests.

  • If one Schema Registry node goes down, another node is elected primary and the cluster auto-recovers.
  • Restart the node, and it will come back as a follower (since a new primary was elected in the meantime).

Suggested Reading