Confluent Developer Marketplace Get started
  1. Home
  2. Platform
  3. Manage Schemas on Confluent Platform
  4. Installing and Configuring Schema Registry on Confluent Platform

Schema Registry Deployment Architectures for Confluent Platform

You can deploy Schema Registry on Confluent Platform with a single primary, using either Kafka or ZooKeeper leader election. You can also set up multiple Schema Registry servers for high availability, where you switch to a secondary Schema Registry cluster if the primary goes down, and for data migration, either one time or as a continuous feed. This section describes these architectures.

Single Primary Architecture

Schema Registry is designed to work as a distributed service that uses a single primary architecture. In this configuration, at most one Schema Registry instance is the primary at any given moment, ignoring pathological “zombie primaries”. Only the primary can publish writes to the underlying Kafka log, but all nodes can directly serve read requests. Secondary nodes serve registration requests indirectly by forwarding them to the current primary and returning the primary’s response. The Kafka group protocol handles primary election. ZooKeeper based primary election was removed in Confluent Platform 7.0.0.

Note

Don’t mix election modes among the nodes in the same cluster. Mixing them leads to multiple primaries and operational issues.

Kafka Coordinator Primary Election

../_images/schema-registry-design-kafka.png

Kafka based Schema Registry

Schema Registry uses Kafka based primary election when kafkastore.connection.url isn’t configured and the Kafka bootstrap brokers are specified in <kafkastore.bootstrap.servers>. The Kafka group protocol chooses one of the primary-eligible nodes (leader.eligibility=true) as the primary. Use Kafka based primary election in all cases. ZooKeeper based leader election was removed in Confluent Platform 7.0.0. For migration steps, see Migration from ZooKeeper primary election to Kafka primary election.

Schema Registry is also designed for multi-colocated configurations. For details, see Multi-Datacenter Setup.

ZooKeeper Primary Election

ZooKeeper leader election was removed in Confluent Platform 7.0.0. Use Kafka leader election instead. For details, see Migration from ZooKeeper primary election to Kafka primary election.

High Availability for Single Primary Setup

Many services in Confluent Platform are effectively stateless, because they store state in Kafka and load it on demand at start-up, and they can redirect requests automatically. Treat these services like any other stateless application, and get high availability effectively for free by deploying multiple instances. Each instance loads all of the Schema Registry state, so any node can serve a READ, and all nodes know how to forward WRITE requests to the primary.

To provide high availability for clients, list multiple Schema Registry instance URLs in the client schema.registry.url property. Clients then use the entire cluster of Schema Registry instances and can fail over to another instance.

This approach also makes it easy to handle changes to the set of servers without reconfiguring and restarting all of your applications. The same strategy applies to REST proxy and Kafka Connect.

A simple multi-node deployment with a few nodes and the single primary election protocol lets Schema Registry fail over easily.

You can also use a single URL for schema.registry.url and still use the entire cluster of Schema Registry instances. However, this configuration doesn’t support failover to different Schema Registry instances in a dynamic DNS or virtual IP setup, because Schema Registry clients see only one schema.registry.url.

Examples and Runbooks

The following sections describe single and multi-datacenter deployments that use Kafka leader election. ZooKeeper leader election was removed in Confluent Platform 7.0.0. To upgrade to Kafka leader election, see Migration from ZooKeeper primary election to Kafka primary election.

Single Datacenter Setup

Within a single datacenter or location, a multi-node, multi-broker cluster provides Kafka data replication across the nodes.

Producers write data to topic partition leaders, and consumers read data from them. Leaders replicate data to followers so that messages are copied to more than one broker.

You can configure parameters on producers and consumers to optimize your single cluster deployment for various goals, including message durability and high availability.

To control when a write is considered successful, set the acks configuration parameter on Kafka producers. For example, setting acks=all requires other brokers in the cluster to acknowledge receiving the data before the leader broker responds to the producer.

If a leader broker fails, the Kafka cluster recovers when a follower broker is elected leader, and client applications can continue to write and read messages through the new leader.

Kafka Election

Important Settings

  • kafkastore.bootstrap.servers: Set this to the primary Kafka cluster (DC A in this example).

  • schema.registry.group.id: Schema Registry uses this setting as the consumer group.id. For a single datacenter setup, make this setting the same for all nodes in the cluster. When set, it overrides group.id for the Kafka group when Kafka is used for leader election. Without this configuration, group.id is “schema-registry”.

  • leader.eligibility: In a single datacenter setup, all Schema Registry instances are local to the Kafka cluster and should have leader.eligibility set to true.

Run Book

If the primary node goes down in a single datacenter Schema Registry deployment, the remaining Schema Registry instances can continue to serve requests.

  • If one Schema Registry node goes down, the cluster elects another node as leader and recovers automatically.

  • Restart the node. It comes back as a follower, because the cluster elected a new leader in the meantime.

Multi-Datacenter Setup

Spanning multiple datacenters (DCs) with your Confluent Schema Registry synchronizes data across sites, further protects against data loss, and reduces latency. In the preferred multi-datacenter deployment, one datacenter is the “primary” and all others are “secondary”. If the “primary” datacenter fails and is unrecoverable, manually designate a “secondary” datacenter as the new “primary” by following the steps in the Run Book later in this section.

Kafka Election

Recommended Deployment
../_images/multi-dc-setup-kafka.png

Multi datacenter with Kafka based primary election

The diagram shows two datacenters, DC A and DC B. Either could be on-premises, in Confluent Cloud, or part of a bridge to cloud solution. Each datacenter has its own Apache Kafka® cluster, ZooKeeper cluster, and Schema Registry.

The Schema Registry nodes in both datacenters link to the primary Kafka cluster in DC A, and the secondary datacenter (DC B) forwards Schema Registry writes to the primary (DC A). Schema Registry nodes and hostnames must be addressable and routable across the two sites to support this configuration.

Schema Registry instances in DC B have leader.eligibility set to false, meaning that none can be elected leader during steady state operation with both datacenters online.

To protect against complete loss of DC A, run Replicator local to the target cluster (DC B) to replicate Kafka cluster A (the source) to Kafka cluster B (the target).

In this active-passive setup, Replicator runs in one direction, copying Kafka data and configurations from the active DC A to the passive DC B. The Schema Registry instances in both datacenters point to the internal _schemas topic in DC A. For disaster recovery, you must replicate the internal schemas topic itself, because if DC A goes down, the system fails over to DC B, and DC B needs a copy of the _schemas topic.

Tip

This failover scenario doesn’t require the same configuration as schema migration. Don’t set schema.registry.topic or schema.subject.translator.class, as you would for a schema migration.

Producers write data only to the active cluster. Depending on the overall design, consumers can read data from the active cluster only, which leaves the passive cluster for disaster recovery, or from both clusters to optimize reads on a geo-local cache.

In the event of a partial or complete disaster in one datacenter, applications can fail over to the secondary datacenter.

ACLs and Security

In a multi-DC setup with ACLs enabled, you must replicate the schemas ACL topic.

During an outage, the ACLs are cached along with the schemas, so Schema Registry continues to run READs with ACLs if the primary Kafka cluster goes down.

Important Settings

  • kafkastore.bootstrap.servers: Set this to the primary Kafka cluster (DC A in this example).

  • schema.registry.group.id: Use this setting to override the group.id for the Kafka group used when Kafka is used for leader election. Without this configuration, group.id is “schema-registry”. If you run more than one Schema Registry cluster against a single Kafka cluster, make this setting unique for each cluster.

  • leader.eligibility: A Schema Registry server with leader.eligibility set to false is guaranteed to remain a follower during any leader election. Schema Registry instances in a “secondary” datacenter should have this set to false, and Schema Registry instances local to the shared Kafka (primary) cluster should have this set to true.

Hostnames must be reachable and resolve across datacenters to support forwarding of new schemas from DC B to DC A.

Setup

If Schema Registry is already running, follow these steps to add Schema Registry instances in a new “secondary” datacenter (call it DC B):

  1. In DC B, set unclean.leader.election.enable to false in Kafka.

  2. In DC B, run Replicator with Kafka in the “primary” datacenter (DC A) as the source and Kafka in DC B as the target.

  3. In the Schema Registry configuration files in DC B, set kafkastore.bootstrap.servers to point to the Kafka cluster in DC A, and set leader.eligibility to false.

  4. Start the new Schema Registry instances with these configurations.

Run Book

If you lose the “primary” datacenter in a multi-datacenter Schema Registry deployment, the remaining Schema Registry instances in the “secondary” datacenter can continue to serve any request that doesn’t result in a write to Kafka. This includes GET requests on existing IDs and POST requests on schemas already in the registry. These instances can’t register new schemas.

  • If possible, revive the “primary” datacenter by starting Kafka and Schema Registry as before.

  • If you must designate a new datacenter (call it DC B) as “primary”, reconfigure kafkastore.bootstrap.servers in DC B to point to its local Kafka cluster, and update the Schema Registry configuration files to set leader.eligibility to true.

  • Restart the Schema Registry instances with these new configurations in a rolling fashion.

Multi-Cluster Schema Registry

In the previous disaster recovery scenarios, a single Schema Registry typically serves multiple environments, with each environment potentially containing multiple Kafka clusters.

Starting with version 5.4.1, Confluent Platform can run multiple schema registries and associate a unique Schema Registry with each Kafka cluster in multi-cluster environments. The primary goal of these deployments is to scale by adding special-purpose registries that support governance across diverse and massive datasets in large organizations. They don’t target disaster recovery.

For details about this configuration, see Enabling Multi-Cluster Schema Registry.

Related Content