Deploy Schema Registry in Production on Confluent Platform

Review these key considerations before deploying Schema Registry to production. Topics include hardware sizing, JVM tuning, network configuration, and Kafka-based primary election.

Hardware

Production deployments of Schema Registry differ from a development setup on a laptop or a small cluster of machines. Use the following recommendations as guidance. None of them is a hard-and-fast rule.

Memory

Schema Registry uses Kafka as a commit log to store all registered schemas durably, and it maintains a few in-memory indices to speed up schema lookups. A conservative upper bound on the number of unique schemas registered in a large data-oriented company like LinkedIn is around 10,000. Assuming roughly 1,000 bytes of heap overhead per schema on average, a 1 GB heap is more than sufficient.

CPUs

CPU usage in Schema Registry is light. The most computationally intensive task is checking the compatibility of two schemas, an infrequent operation that occurs primarily when a new schema version is registered under a subject.

If you need to choose between faster CPUs or more cores, choose more cores. The extra concurrency from multiple cores far outweighs a slightly faster clock speed.

Disks

Schema Registry doesn’t have any disk-resident data. It uses Kafka as a commit log to store all schemas durably and holds in-memory indices of all schemas. The only disk usage comes from storing the log4j logs.

Network

A fast, reliable network is important to performance in a distributed system. Low latency helps nodes communicate easily, and high bandwidth helps shard movement and recovery. Modern data-center networking (1 GbE, 10 GbE) is sufficient for the vast majority of clusters.

Avoid clusters that span multiple data centers, even if the data centers are close together, and especially avoid clusters that span large geographic distances.

Larger latencies tend to worsen problems in distributed systems and make debugging and resolution harder.

The connection between data centers is often neither robust nor low latency, and network failures can happen. For a multi-datacenter setup, see Multi-Datacenter Setup.

JVM

For the Java and JDK versions to use, see Java. Run the latest version of the JDK with the Garbage First (G1) collector, because older freely available versions have disclosed security vulnerabilities.

Use the following garbage collector (GC) tuning:

-Xms1g -Xmx1g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 \
       -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M \
       -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80

Important Configuration Options

Change the following configurations for production environments. These options depend on your cluster layout.

For multi-cluster deployments, configure Schema Registry to use Kafka-based primary election as described in the following sections. For full descriptions of these properties, see Schema Registry Configuration Reference for Confluent Platform.

Kafka based primary election

Kafka based primary election is available in Confluent Platform 4.0 and later and is the preferred method for leader election. To use it, configure the kafkastore.bootstrap.servers setting.

kafkastore.bootstrap.servers

with listener.name.<name>. to apply that setting only to the listener named <name>. The prefixed form overrides the unprefixed form for that listener.

The inter.instance.listener.name setting names the inter-instance listener. Use this prefix to give that listener different SSL settings than the public listener. For example, configure the inter-instance listener with mTLS using a separate keystore and truststore.

listeners

kafkastore.sasl.kerberos.min.time.before.relogin

The login time between refresh attempts.

Type: long
Default: 60000
Importance: low

host.name

The advertised host name. Make sure to set this if running Schema Registry with multiple nodes. This name is always used in the endpoint for communication between Schema Registry instances, even if inter.instance.listener.name is specified, or does not match any listener. (The schema, port and TLS configurations will continue to be picked based on the inter.instance.listener.name config, if it is set.)

Type: string
Default: “192.168.50.1”
Importance: high

Important

If host.name is set in a multi Schema Registry node environment, it must resolve to a valid location or URL from the other Schema Registry instance(s) to ensure communication between the nodes.

Note

Configure min.insync.replicas on the Kafka server for the internal _schemas topic that stores all registered schemas to be higher than 1. For example, if kafkastore.topic.replication.factor is 3, set min.insync.replicas on the Kafka server for the kafkastore.topic to 2. This ensures that a schema registration write is considered durable if it’s committed on at least 2 of 3 replicas. It’s also best to set unclean.leader.election.enable to false so that a replica outside of the ISR is never elected leader, which could result in data loss.

For the full set of configuration options, see Schema Registry Configuration Reference for Confluent Platform.

ZooKeeper based primary election

Important

ZooKeeper leader election and the use of kafkastore.connection.url for it were removed in Confluent Platform 7.0.0. Use Kafka leader election instead. For details, see Migration from ZooKeeper primary election to Kafka primary election.

To migrate from ZooKeeper based to Kafka based primary election, see Migration from ZooKeeper primary election to Kafka primary election.

Don’t Modify These Storage Settings

Schema Registry stores all schemas in a Kafka topic defined by kafkastore.topic. This topic acts as the commit log and source of truth for the Schema Registry database, so writes to it must be durable, and it must be a compacted topic to avoid data loss. Schema Registry ships with good defaults for all settings that affect the durability of writes to the Kafka-based commit log. If in doubt, leave these settings alone. If you must create the topic manually, use the following example configuration:

# kafkastore.topic=_schemas
  bin/kafka-topics --create --bootstrap-server localhost:9092 --topic _schemas --replication-factor 3 --partitions 1 --config cleanup.policy=compact

kafkastore.topic

The durable single partition topic that acts as the durable log for the data. This topic must be compacted to avoid losing data due to retention policy.

Type: string
Default: “_schemas”
Importance: high

kafkastore.topic.replication.factor

The desired replication factor of the schema topic. The actual replication factor will be the smaller of this value and the number of live Kafka brokers.

Type: int
Default: 3
Importance: high

kafkastore.init.timeout.ms

The timeout for initialization of the Kafka store, including creation of the Kafka topic that stores schema data.

Type: int
Default: 60000
Importance: medium

Migration from ZooKeeper primary election to Kafka primary election

This section explains how to reconfigure Schema Registry for Kafka leader election.

Important

ZooKeeper leader election and the use of kafkastore.connection.url for it were removed in Confluent Platform 7.0. Use Kafka leader election instead.

To migrate from ZooKeeper based to Kafka based primary election, make the following configuration changes on all Schema Registry nodes:

Remove kafkastore.connection.url.
Remove schema.registry.zk.namespace if it is configured.
Configure kafkastore.bootstrap.servers.
Configure schema.registry.group.id if you originally had schema.registry.zk.namespace for multiple Schema Registry clusters.

If both kafkastore.connection.url and kafkastore.bootstrap.servers are configured, Schema Registry uses Kafka for leader election.

Downtime for Writes

To migrate from ZooKeeper based primary election to Kafka based primary election, follow these steps. Schema Registry is unavailable for writes for a brief time during the migration.

Make the preceding configuration changes on each node, and set leader.eligibility to false on all nodes.
Perform a rolling bounce of all nodes.
Set leader.eligibility to true on the nodes that can be primary-eligible, and bounce them.

Complete Downtime

For a simpler migration, take Schema Registry down temporarily. Shut down all nodes, and then start them again with the new configuration.

Backup and Restore

As discussed in Kafka backend, Schema Registry appends all schemas, subject and version metadata, ID metadata, and compatibility settings as messages to a special Kafka topic, <kafkastore.topic> (default _schemas). This topic is a common source of truth for schema IDs. If an unexpected event makes the topic inaccessible, you can restore it from a backup so that consumers can continue to read Kafka messages that were sent in the Avro format.

Back up the <kafkastore.topic> as a best practice. You have three options, described in the following sections.

Backups using Replicator

If you already have a multi-datacenter Kafka deployment, you can back up the <kafkastore.topic> (_schemas) to another Kafka cluster using Confluent Replicator.

Backups using a sink connector

Alternatively, use a sink-to-storage connector, like the Kafka Connect Amazon S3 sink connector, to back up your data to separate storage, such as AWS S3, and keep it continuously updated. Copy the <kafkastore.topic> (_schemas) as raw data (bytes). With the Connect S3 sink connector, you can do this in either of the following ways:

To store Kafka records and the _schemas topic in S3, use the ByteArrayConverter to back up the _schemas topic as raw data. Then use the topics or topics.regex configuration property to list other Kafka topics to back up. For descriptions of these properties, see Kafka Sink Connector Configuration Reference for Confluent Platform.
To store Kafka records in JSON, Avro, or Parquet, use one S3 sink connector to back up a list of Kafka topics. Then create a second S3 sink connector to copy the _schemas topic to S3 using Bytes format.

Backups using command line tools

As another option, use Kafka command line tools to periodically save the contents of the topic to a file. The following examples assume that <kafkastore.topic> has its default value, _schemas.

To back up the topic, use kafka-console-consumer to capture messages from the schemas topic to a file called schemas.log, and then save the file outside the Kafka cluster.

bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic _schemas --from-beginning --property print.key=true --timeout-ms 1000 1> schemas.log

To restore the topic, use kafka-console-producer to write the contents of schemas.log to a new schemas topic. This example uses the new topic name _schemas_restore. Whether you use a new topic name or the old one (_schemas), set <kafkastore.topic> accordingly.

bin/kafka-console-producer --bootstrap-server localhost:9092 --topic _schemas_restore --property parse.key=true < schemas.log