Use Broker-Side Schema ID Validation on Confluent Cloud

Schema ID Validation enables the broker to verify that data produced to a Kafka topic uses a valid schema ID in Schema Registry. The schema ID must belong to a subject that matches the subject naming strategy. To learn more about subjects and topics, see Schemas, subjects, and topics.

Schema ID Validation checks that Schema Registry has the schema ID from the Wire format under a valid subject. It doesn’t introspect the message data.

You must use a serializer and deserializer (SerDes) that respect the Wire format, or a Confluent-supported serde. For more information, see Formats, Serializers, and Deserializers.

Limitations

Schema ID Validation doesn’t reject tombstone records, which have a null value, or messages with a null key, even if the record has no schema ID. This design choice supports effective data management and deletion in compacted topics. For details, see Messages with null keys or values pass schema validation.

Prerequisites

  • Schema ID Validation on Confluent Cloud is available only on Dedicated clusters through the hosted Schema Registry. Confluent Cloud brokers can’t use self-managed Schema Registry instances. Confluent Enterprise supports Schema ID Validation for on-premises deployments. To learn more, see Schema validation.

  • You must have Schema Registry enabled for the environment in which you use Schema ID Validation.

  • Schema ID Validation works at the environment level. All Dedicated clusters in the same environment share a Schema Registry. Clusters can’t see schemas in other environments.

Schema ID Validation configuration options on a topic

Set Schema ID Validation at the topic level with the following properties:

Property

Description

confluent.key.schema.validation

When set to true, enables schema ID validation on the message key. The default is false.

confluent.value.schema.validation

When set to true, enables schema ID validation on the message value. The default is false.

confluent.key.subject.name.strategy

Subject name strategy for the message key. The default is io.confluent.kafka.serializers.subject.TopicNameStrategy.

confluent.value.subject.name.strategy

Subject name strategy for the message value. The default is io.confluent.kafka.serializers.subject.TopicNameStrategy.

confluent.schema.validation.context.name

Schema context that the broker searches when it validates the schema ID of the message key and value. The default is default.

Tip

  • Key and value schema validation are independent. You can enable key validation, value validation, or both.

  • The subject naming strategy applies only when Schema ID Validation is enabled.

Update the Confluent CLI

Run confluent update to update the Confluent CLI to the latest version. For the available commands, see the Confluent CLI command reference.

To learn more about the Confluent CLI and migration paths, see Install Confluent CLI, Migrate to the latest version of the Confluent CLI, and Run multiple CLIs in parallel.

Enable Schema ID Validation from the Confluent CLI

Enable Schema ID Validation on a topic when you create the topic, or update an existing topic, on a Dedicated cluster. Use the following command syntax:

confluent kafka topic <create|update> <topic-name> --config confluent.<key|value>.schema.validation=true

For example, the following command creates a topic named flights with Schema ID Validation enabled on the value schema:

confluent kafka topic create flights --config confluent.value.schema.validation=true

With this configuration, if a producer sends a message to flights whose value has no valid schema, the broker returns an error to the producer and discards the message. If any message in a batch is invalid, the broker discards the entire batch.

The default subject naming strategy is io.confluent.kafka.serializers.subject.TopicNameStrategy. You can change the strategy for the message key, the message value, or both. For example, the following command sets the subject naming strategy on the topic flights to io.confluent.kafka.serializers.subject.RecordNameStrategy:

confluent kafka topic update flights --config confluent.value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy

The confluent.value.subject.name.strategy property accepts the following naming strategies:

Strategy

Description

TopicNameStrategy

Derives the subject name from the topic name. This strategy is the default.

RecordNameStrategy

Derives the subject name from the record name. This strategy groups logically related events that can have different data structures under one subject.

TopicRecordNameStrategy

Derives the subject name from the topic name and the record name. This strategy groups logically related events that can have different data structures under one subject.

Note

The full class name of each strategy is the strategy name prefixed with io.confluent.kafka.serializers.subject..

Enable Schema ID Validation on a topic from the Confluent Cloud Console

To set Schema ID Validation on a topic from the Cloud Console:

  1. Navigate to a topic.

  2. Click the Configuration tab.

  3. Click Edit Settings.

  4. Click Switch to expert mode.

  5. In Expert mode, change confluent.value.schema.validation, confluent.key.schema.validation, or both from false to true.

    Expert mode on the Configuration tab of the flights topic, with confluent_value_schema_validation set to true

    Tip

    If the Schema ID Validation properties don’t appear in Expert mode, confirm that the cluster is a Dedicated cluster. Schema ID Validation is available only on Dedicated clusters.

  6. Optional: to change the naming strategy for the message key, the message value, or both, edit confluent.key.subject.name.strategy or confluent.value.subject.name.strategy in Expert mode.

  7. Click Save changes.

To disable Schema ID Validation, set the same properties to false.

Schema ID Validation demo

Follow this demo to test Schema ID Validation.

  1. Create a test topic named players-maple from the Confluent Cloud Console or the Confluent CLI. Leave the Schema ID Validation setting unspecified so that the topic defaults to false.

    To create the topic with the Confluent CLI, run this command:

    confluent kafka topic create players-maple
    

    The command creates a topic with no broker validation on records produced to it, which is what the first part of the demo needs.

  2. Open a new command window for the producer and log in to Confluent Cloud on the same environment and cluster.

  3. Produce a record, serialized with the default string serializer, to players-maple:

    confluent kafka topic produce players-maple --parse-key=true --delimiter=,
    

    The command succeeds because Schema ID Validation is disabled for this topic. If Schema ID Validation were enabled, the broker would reject the record.

    Type your first message at the producer prompt:

    1,Pierre
    

    Keep the producer session running.

  4. Open a new command window for the consumer and log in to Confluent Cloud on the same environment and cluster.

  5. Read the messages:

    confluent kafka topic consume players-maple --from-beginning --print-key=true
    

    The output of this command is 1    Pierre.

    Keep the consumer session running.

  6. Set Schema ID Validation for the topic players-maple to true:

    confluent kafka topic update players-maple --config confluent.value.schema.validation=true
    

    Tip

    You can also update this setting in Expert mode for the players-maple topic configuration in the Confluent Cloud Console.

  7. Return to the producer session and type a second message at the prompt:

    2,Frederik
    

    The producer returns an error because Schema ID Validation is enabled and the message doesn’t contain a schema ID: Error: producer has detected an INVALID_RECORD error for topic players-maple

    To disable Schema ID Validation, run the same update command with false. Then resend the message or produce a similar message, such as 3,Ben. The broker accepts the message.

    The Confluent Cloud Console also shows the messages that the producer sent successfully under Topics > players-maple > Messages. To see earlier messages, select a partition or jump to a timestamp.

What Schema ID Validation checks and how it works

Schema ID Validation checks each message for the following:

  • The message has a schema ID, which indicates the message has an associated schema.

  • The schema matches the topic.

The demo confirms that Schema ID Validation works, using the Confluent CLI.

In practice, a client application sends an Avro object, a Protobuf object, or a Jackson-serializable POJO. The serializer derives the schema from the object and sends it to Schema Registry. Schema Registry checks whether the schema exists in the subject. If it does, Schema Registry uses the schema ID of that version. If it doesn’t, Schema Registry returns an error when the client has auto schema registration set to false and registers the schema when the client has it set to true.

Set auto schema registration in the client application. By default, clients register new schemas automatically. Disabling auto schema registration on clients is typical in production environments. To learn more, see Disabling Auto Schema Registration in the Confluent Platform documentation.

Use a specific schema context with schema validation

Set confluent.schema.validation.context.name to a specific schema context, such as .mycontext or mycontext. The broker then searches that context for subjects when it validates the schema ID of message keys and values. If the context doesn’t exist in Schema Registry, schema validation fails and the broker rejects messages.

The default is default. The behavior of the default depends on the subject name strategy that Schema ID Validation uses.

With TopicNameStrategy, the broker searches for the subject name across all schema contexts:

  • If the subject exists in the default schema context, the broker uses it to perform schema validation.

  • If the subject doesn’t exist in the default schema context, the broker searches all available contexts and uses the first subject name it finds.

With RecordNameStrategy or TopicRecordNameStrategy, the broker searches for the subject only in the default context. If the broker doesn’t find the subject in the default context, schema validation fails. Make sure that the producers refer to the correct schema context.

To configure the schema context in clients, see Specifying a context name for clients.

Set this property with the Confluent CLI, Terraform, or the Schema Registry APIs. The Confluent Cloud Console doesn’t support setting this property.

Troubleshoot

If the demo doesn’t work, or you can’t see confluent.value.schema.validation, check the following:

  • Schema ID Validation is available only on Dedicated clusters through the hosted Schema Registry. Confirm that the cluster is a Dedicated Kafka cluster, not a Basic, Standard, or Enterprise cluster.

  • Schema ID Validation requires Schema Registry in the environment. Confirm that you have enabled |sr| for the environment in which you use Schema ID Validation.

Messages with null keys or values pass schema validation

A Kafka message with a null value passes schema validation when confluent.value.schema.validation is enabled, even though a null value doesn’t follow the Wire format. A Kafka message with a null key passes schema validation when confluent.key.schema.validation is enabled, for the same reason.