Use Broker-Side Schema ID Validation on Confluent Cloud
Schema ID Validation enables the broker to verify that data produced to a Kafka topic uses a valid schema ID in Schema Registry. The schema ID must belong to a subject that matches the subject naming strategy. To learn more about subjects and topics, see Schemas, subjects, and topics.
Schema ID Validation checks that Schema Registry has the schema ID from the Wire format under a valid subject. It doesn’t introspect the message data.
You must use a serializer and deserializer (SerDes) that respect the Wire format, or a Confluent-supported serde. For more information, see Formats, Serializers, and Deserializers.
Limitations
Schema ID Validation doesn’t reject tombstone records, which have a null value, or messages with a null key, even if the record has no schema ID. This design choice supports effective data management and deletion in compacted topics. For details, see Messages with null keys or values pass schema validation.
Prerequisites
Schema ID Validation on Confluent Cloud is available only on Dedicated clusters through the hosted Schema Registry. Confluent Cloud brokers can’t use self-managed Schema Registry instances. Confluent Enterprise supports Schema ID Validation for on-premises deployments. To learn more, see Schema validation.
You must have Schema Registry enabled for the environment in which you use Schema ID Validation.
Schema ID Validation works at the environment level. All Dedicated clusters in the same environment share a Schema Registry. Clusters can’t see schemas in other environments.
Schema ID Validation configuration options on a topic
Set Schema ID Validation at the topic level with the following properties:
Property |
Description |
|---|---|
|
When set to |
|
When set to |
|
Subject name strategy for the message key. The default is |
|
Subject name strategy for the message value. The default is |
|
Schema context that the broker searches when it validates the schema ID of the message key and value. The default is |
Tip
Key and value schema validation are independent. You can enable key validation, value validation, or both.
The subject naming strategy applies only when Schema ID Validation is enabled.
Update the Confluent CLI
Run confluent update to update the Confluent CLI to the latest version.
For the available commands, see the
Confluent CLI command reference.
To learn more about the Confluent CLI and migration paths, see Install Confluent CLI, Migrate to the latest version of the Confluent CLI, and Run multiple CLIs in parallel.
Enable Schema ID Validation from the Confluent CLI
Enable Schema ID Validation on a topic when you create the topic, or update an existing topic, on a Dedicated cluster. Use the following command syntax:
confluent kafka topic <create|update> <topic-name> --config confluent.<key|value>.schema.validation=true
For example, the following command creates a topic named flights with
Schema ID Validation enabled on the value schema:
confluent kafka topic create flights --config confluent.value.schema.validation=true
With this configuration, if a producer sends a message to flights whose
value has no valid schema, the broker returns an error to the producer and
discards the message. If any message in a batch is invalid, the broker
discards the entire batch.
The default subject naming strategy is
io.confluent.kafka.serializers.subject.TopicNameStrategy. You can change
the strategy for the message key, the message value, or both. For example, the
following command sets the subject naming strategy on the topic flights to
io.confluent.kafka.serializers.subject.RecordNameStrategy:
confluent kafka topic update flights --config confluent.value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
The confluent.value.subject.name.strategy property accepts the following
naming strategies:
Strategy |
Description |
|---|---|
|
Derives the subject name from the topic name. This strategy is the default. |
|
Derives the subject name from the record name. This strategy groups logically related events that can have different data structures under one subject. |
|
Derives the subject name from the topic name and the record name. This strategy groups logically related events that can have different data structures under one subject. |
Note
The full class name of each strategy is the strategy name prefixed
with io.confluent.kafka.serializers.subject..
Enable Schema ID Validation on a topic from the Confluent Cloud Console
To set Schema ID Validation on a topic from the Cloud Console:
Navigate to a topic.
Click the Configuration tab.
Click Edit Settings.
Click Switch to expert mode.
In Expert mode, change
confluent.value.schema.validation,confluent.key.schema.validation, or both fromfalsetotrue.
Tip
If the Schema ID Validation properties don’t appear in Expert mode, confirm that the cluster is a Dedicated cluster. Schema ID Validation is available only on Dedicated clusters.
Optional: to change the naming strategy for the message key, the message value, or both, edit
confluent.key.subject.name.strategyorconfluent.value.subject.name.strategyin Expert mode.Click Save changes.
To disable Schema ID Validation, set the same properties to false.
Schema ID Validation demo
Follow this demo to test Schema ID Validation.
Create a test topic named
players-maplefrom the Confluent Cloud Console or the Confluent CLI. Leave the Schema ID Validation setting unspecified so that the topic defaults tofalse.To create the topic with the Confluent CLI, run this command:
confluent kafka topic create players-maple
The command creates a topic with no broker validation on records produced to it, which is what the first part of the demo needs.
Open a new command window for the producer and log in to Confluent Cloud on the same environment and cluster.
Produce a record, serialized with the default string serializer, to
players-maple:confluent kafka topic produce players-maple --parse-key=true --delimiter=,
The command succeeds because Schema ID Validation is disabled for this topic. If Schema ID Validation were enabled, the broker would reject the record.
Type your first message at the producer prompt:
1,PierreKeep the producer session running.
Open a new command window for the consumer and log in to Confluent Cloud on the same environment and cluster.
Read the messages:
confluent kafka topic consume players-maple --from-beginning --print-key=true
The output of this command is
1 Pierre.Keep the consumer session running.
Set Schema ID Validation for the topic
players-mapletotrue:confluent kafka topic update players-maple --config confluent.value.schema.validation=true
Tip
You can also update this setting in Expert mode for the
players-mapletopic configuration in the Confluent Cloud Console.Return to the producer session and type a second message at the prompt:
2,FrederikThe producer returns an error because Schema ID Validation is enabled and the message doesn’t contain a schema ID:
Error: producer has detected an INVALID_RECORD error for topic players-mapleTo disable Schema ID Validation, run the same update command with
false. Then resend the message or produce a similar message, such as3,Ben. The broker accepts the message.The Confluent Cloud Console also shows the messages that the producer sent successfully under Topics > players-maple > Messages. To see earlier messages, select a partition or jump to a timestamp.
What Schema ID Validation checks and how it works
Schema ID Validation checks each message for the following:
The message has a schema ID, which indicates the message has an associated schema.
The schema matches the topic.
The demo confirms that Schema ID Validation works, using the Confluent CLI.
In practice, a client application sends an
Avro object, a
Protobuf object, or a
Jackson-serializable POJO.
The serializer derives the schema from the object and sends it to Schema Registry. Schema Registry
checks whether the schema exists in the
subject. If it
does, Schema Registry uses the schema ID of that version. If it doesn’t, Schema Registry returns an
error when the client has auto schema registration set to false and
registers the schema when the client has it set to true.
Set auto schema registration in the client application. By default, clients register new schemas automatically. Disabling auto schema registration on clients is typical in production environments. To learn more, see Disabling Auto Schema Registration in the Confluent Platform documentation.
Use a specific schema context with schema validation
Set confluent.schema.validation.context.name to a specific schema context,
such as .mycontext or mycontext. The broker then searches that context
for subjects when it validates the schema ID of message keys and values. If the
context doesn’t exist in Schema Registry, schema validation fails and the broker rejects
messages.
The default is default. The behavior of the default depends on the
subject name strategy that Schema ID Validation uses.
With TopicNameStrategy, the broker searches for the subject name across all
schema contexts:
If the subject exists in the default schema context, the broker uses it to perform schema validation.
If the subject doesn’t exist in the default schema context, the broker searches all available contexts and uses the first subject name it finds.
With RecordNameStrategy or TopicRecordNameStrategy, the broker searches
for the subject only in the default context. If the broker doesn’t find the
subject in the default context, schema validation fails. Make sure that the
producers refer to the correct schema context.
To configure the schema context in clients, see Specifying a context name for clients.
Set this property with the Confluent CLI, Terraform, or the Schema Registry APIs. The Confluent Cloud Console doesn’t support setting this property.
Troubleshoot
If the demo doesn’t work, or you can’t see
confluent.value.schema.validation, check the following:
Schema ID Validation is available only on Dedicated clusters through the hosted Schema Registry. Confirm that the cluster is a Dedicated Kafka cluster, not a Basic, Standard, or Enterprise cluster.
Schema ID Validation requires Schema Registry in the environment. Confirm that you have enabled |sr| for the environment in which you use Schema ID Validation.
Messages with null keys or values pass schema validation
A Kafka message with a null value passes schema validation when
confluent.value.schema.validation is enabled, even though a null value
doesn’t follow the Wire format. A Kafka message with a null key
passes schema validation when confluent.key.schema.validation is enabled,
for the same reason.