Confluent Developer Marketplace Get started
  1. Home
  2. Platform
  3. Manage Schemas on Confluent Platform
  4. Fundamentals
  5. Schema Formats for Schema Registry on Confluent Platform

Formats, Serializers, and Deserializers for Schema Registry on Confluent Platform

Serializers and deserializers (SerDes) convert data between your application’s objects and the bytes Kafka stores, using a format such as Avro, Protobuf, or JSON Schema. These formats and tools apply to Confluent Cloud and Confluent Platform, though developing applications that use them requires Confluent Platform.

Ready to get started?

Schema Registry supports Protocol Buffers and JSON Schema along with Avro, the original default format. Support for these new serialization formats is not limited to Schema Registry, but provided throughout Confluent products. Additionally, Schema Registry is extensible to support adding custom schema formats as schema plugins.

New Kafka serializers and deserializers are available for Protobuf and JSON Schema, along with Avro. The serializers can automatically register schemas when serializing a Protobuf message or a JSON-serializable object. The Protobuf serializer can recursively register all imported schemas, .

The serializers and deserializers are available in multiple languages, including Java, .NET and Python.

Schema Registry supports multiple formats at the same time. For example, you can have Avro schemas in one subject and Protobuf schemas in another. Furthermore, both Protobuf and JSON Schema have their own compatibility rules, so you can have your Protobuf schemas evolve in a backward or forward compatible manner, just as with Avro.

Schema Registry also supports for schema references in Protobuf by modeling the import statement.

Supported formats

Schema Registry supports the following schema formats, with serializers, deserializers, and command line tools for each format.

Format

Producer

Consumer

Avro

io.confluent.kafka.serializers.KafkaAvroSerializer

io.confluent.kafka.serializers.KafkaAvroDeserializer

Protobuf

io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer

io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer

JSON Schema

io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer

io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer

Note

JSON and PROTOBUF_NOSR aren’t supported. For details, see What’s supported in the Schema Registry overview.

Use the serializer and deserializer for your schema format. Specify the serializer in the Kafka producer code to send messages. Specify the deserializer in the Kafka consumer code to read messages.

Subject name strategies

The subject name strategy sets the subject under which a serializer registers a schema. The Protobuf and JSON Schema serializers and deserializers support many of the same configuration properties as the Avro equivalents, including subject name strategies for the key and value. With RecordNameStrategy or TopicRecordNameStrategy, the subject name is:

  • For Avro, the record full name, which is the namespace and the record name.

  • For Protobuf, the message name.

  • For JSON Schema, the title.

Using RecordNameStrategy with Protobuf and JSON Schema requires additional configuration. The following sections cover the configuration, examples, and command line testing utilities for each format:

Produce and consume examples

Produce and consume examples for each Schema Registry SerDes are available in confluentinc/confluent-kafka-go/examples.

Schema registration

The serializers and Kafka Connect converters for all supported schema formats automatically register schemas by default. The Protobuf serializer recursively registers all referenced schemas separately.

Add a format using a schema plugin

Schema Registry lets you add new schema formats with schema plugins. The Avro format uses an Avro schema plugin as well. For details, see Schema format extensibility.

Client support notes

Client versions for Schema Registry track Confluent Platform versions, whether you use Confluent Cloud or Confluent Platform.

Schema Registry clients 7.7.4 and later automatically retry requests that fail with a 429 exception. A 429 exception means the Schema Registry server rate limited the request. When a client exceeds the set rate limit, the server responds with an HTTP 429 status code that tells the client to slow down its requests. Schema Registry clients built with Confluent Platform versions earlier than 7.7.4 don’t retry 429 exceptions automatically.

Schema references

Confluent Platform 5.5.0 and later and Confluent Cloud fully support schema references, which let a schema refer to other schemas. Avro, JSON Schema, and Protobuf all support schema references. Protobuf references use the import statement, and JSON Schema references use the $ref field.

A schema reference consists of the following:

  • A name for the reference. For Avro, the name is the fully qualified schema name. For JSON Schema, the name is a URL. For Protobuf, the name identifies another Protobuf file.

  • A subject for the referenced schema.

  • A version of the referenced schema under that subject.

When you register a schema that has references, you must provide the references. Typically, you register the referenced schemas first. Then use their subjects and versions when you register the schema that references them.

When you retrieve a schema that has references from Schema Registry, Schema Registry also retrieves the referenced schemas if needed.

For examples of schema references in each format, see:

Multiple event types in the same topic

Use schema references to combine multiple event types in the same topic and still maintain subject-topic constraints.

For examples in each format, see:

Schema reference endpoints

Two Schema Registry endpoints show how schemas relate to each other and to subjects. For details, see the Schema Registry API.

  • List the IDs of schemas that reference the schema with the given subject and version:

    GET /subjects/{subject}/versions/{version}/referencedby

  • List all subject-version pairs that use the given schema ID:

    GET /schemas/ids/{id}/versions

Schema format extensibility

Schema format extensibility lets you define and integrate custom schema formats with schema plugins.

Note

Schema format extensibility is available only on Confluent Platform. Schema Registry on Confluent Cloud doesn’t support custom schema plugins.

By default, Schema Registry loads schema plugins for Avro, Protobuf, and JSON Schema. When you use the REST API, specify the schema type as AVRO, PROTOBUF, or JSON, respectively.

To create a custom schema plugin, implement the SchemaProvider and ParsedSchema interfaces. Then load the plugin into Schema Registry:

  1. Place the plugin JARs on the CLASSPATH.

  2. Set the following Schema Registry configuration property to a comma-separated list of the plugin provider classes:

    schema.providers=com.acme.MySchemaProvider

Don’t include the schema plugins for Avro, Protobuf, or JSON Schema. Schema Registry always loads them.

Subject name strategy

A serializer registers a schema in Schema Registry under a subject name, which defines a namespace in the registry. The subject scopes the following:

  • Compatibility checks apply per subject.

  • Versions belong to a subject.

  • When a schema evolves, it stays associated with the same subject and gets a new schema ID and version.

Subject name strategy overview

The subject name depends on the subject name strategy. The full class name of each strategy is the strategy name prefixed by io.confluent.kafka.serializers.subject.. The following table describes the strategies:

Strategy

Description

TopicNameStrategy

Derives the subject name from the topic name. This strategy is the default.

RecordNameStrategy

Derives the subject name from the record name and lets you group logically related events that can have different data structures under a topic.

TopicRecordNameStrategy

Derives the subject name from the topic name and the record name. This strategy lets you group logically related events that can have different data structures under a topic.

Note

The subject name strategy configured on a topic in the broker for schema ID validation doesn’t propagate to clients. Configure the subject name strategy separately in the clients.

Group by topic or other relationships

The default strategy, TopicNameStrategy, names the schema based on the topic name. It implicitly requires all messages in the same topic to conform to the same schema, because a new record type could break compatibility checks on the topic. This strategy works well when grouping messages by topic name makes sense, such as aggregating logged activities or stream processing website comment threads.

The non-default strategies, RecordNameStrategy and TopicRecordNameStrategy, support schema management for use cases where grouping by topic isn’t optimal. For example, a single topic can have records that use multiple schemas. These strategies are useful when your data represents a time-ordered sequence of events and the messages have different data structures. In this case, it’s more useful to keep related messages together in order, regardless of topic name, when the messages play a part in a chain of events. For example, a financial service that tracks a customer account might record opening checking and savings accounts, making a deposit, making a withdrawal, applying for a loan, and getting approval.

How the naming strategies work

The following table compares the strategies.

Behavior

TopicNameStrategy

RecordNameStrategy

TopicRecordNameStrategy

Subject format

<topic name> plus “-key” or “-value” depending on configuration

<fully qualified record name>

<topic name>-<fully qualified record name>

Unique subject per topic

Yes

No

Yes

Schema Registry checks compatibility across all schemas in a topic

Yes

No. Schema Registry checks compatibility of any occurrences of the same record name across all topics.

Yes. Different topics can contain mutually incompatible versions of the same record name, because the compatibility check is scoped to a particular record name within a particular topic.

Multiple topics can have records with the same schema

Yes

Yes

Yes

Multiple subjects can have schemas with the same schema ID if the schemas are identical

Yes

Yes

Yes

A single topic can have multiple schemas for the same record type, such as during schema evolution

Yes

Yes

Yes

A single topic can have multiple record types

Not generally, because a new record type could break Schema Registry compatibility checks on the topic.

Yes

Yes

Requires the client application to change a setting

No. This strategy is already the default for all clients.

Yes

Yes

You can reuse the same subject for replicated topics that were renamed, such as when Replicator is configured with topic.rename.format

No. You must register the subject manually with the new topic name.

Yes

No. You must register the subject manually with the new topic name.

Limitations

  • ksqlDB uses only the default TopicNameStrategy and doesn’t support multiple schemas in a single topic.

  • The Confluent Cloud Console and Confluent Platform Control Center view and edit schemas only for subjects that use the default TopicNameStrategy. From the topic tabs, they show only schemas linked with subjects derived from TopicNameStrategy. Schema lookups return only subjects named <topic name>-key or <topic name>-value. For example, if a schema subject is named my-first-topic-value, the lookup checks for a topic named my-first-topic. If the lookup finds the topic, it lists the topic as a user of the schema. Lookups don’t find subjects that use other naming strategies.

  • Schema Registry allows a / in a subject name, but it’s cumbersome to use with the Schema Registry Maven Plugin for Confluent Platform. The plugin requires renaming the XML tags with _x2F escape sequences to replace each /. Even then, the / can cause integration issues with topic names and some schema URIs. As a best practice, use ., -, or _ as separator characters.

  • Escape any natural _x occurrence in a subject name, such as my_xyz_subject. The XML parser interprets _x as the start of an escape sequence, such as _xyz. To make the Maven plugin interpret the name correctly, escape _x with _x5F, as in my_x5Fxyz_subject.

Subject name strategy configuration properties

The Kafka serializers use TopicNameStrategy by default to determine the subject name when registering a schema. To change this behavior, use the following configuration properties:

key.subject.name.strategy

Determines how to construct the subject name under which Schema Registry registers the key schema.

Set this to any implementation of io.confluent.kafka.serializers.subject.strategy.SubjectNameStrategy. By default, the subject is <topic>-key. Specifying an implementation of io.confluent.kafka.serializers.subject.SubjectNameStrategy is deprecated as of 4.1.3 and can degrade performance.

  • Type: class

  • Default: class io.confluent.kafka.serializers.subject.TopicNameStrategy

  • Importance: medium

value.subject.name.strategy

Determines how to construct the subject name under which Schema Registry registers the value schema.

Set this to any implementation of io.confluent.kafka.serializers.subject.strategy.SubjectNameStrategy. By default, the subject is <topic>-value. Specifying an implementation of io.confluent.kafka.serializers.subject.SubjectNameStrategy is deprecated as of 4.1.3 and can degrade performance.

  • Type: class

  • Default: class io.confluent.kafka.serializers.subject.TopicNameStrategy

  • Importance: medium

The following strategies are also available:

io.confluent.kafka.serializers.subject.RecordNameStrategy

Registers the schema for any record type published to Kafka under the fully qualified record name, regardless of the topic. This strategy lets a topic contain a mixture of record types because Schema Registry performs no intra-topic compatibility checking. Schema Registry checks compatibility of any occurrences of the same record name across all topics.

io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

Registers the schema for any record type published to the Kafka topic <topicName> under the subject name <topicName>-<recordName>, where <recordName> is the fully qualified record name. This strategy lets a topic contain a mixture of record types because Schema Registry performs no intra-topic compatibility checking. Different topics can contain mutually incompatible versions of the same record name, because Schema Registry checks compatibility only for a particular record name within a particular topic.

Protobuf is the only format that auto-registers schema references. For that reason, Protobuf has an additional configuration property that sets the naming strategy for auto-registered schema references. For Avro and JSON Schema, you typically register references manually and choose the subject name yourself. To change the Protobuf behavior, use the following configuration property:

reference.subject.name.strategy

Set this to any implementation of io.confluent.kafka.serializers.subject.strategy.ReferenceSubjectNameStrategy. The default is DefaultReferenceSubjectNameStrategy, which uses the reference name as the subject.

  • Type: class

  • Default: class io.confluent.kafka.serializers.subject.strategy.DefaultReferenceSubjectNameStrategy

  • Importance: medium

Handle differences between preregistered and client-derived schemas

Use auto.register.schemas, use.latest.version, and latest.compatibility.strict to handle differences between preregistered and client-derived schemas. You can configure these properties in any client that uses a Schema Registry serializer, including producers, Kafka Streams applications, and Connect. For connector-specific details, see Kafka Connect converters. For the full reference, see Configuration Options.

  • auto.register.schemas: whether the serializer attempts to register the schema with Schema Registry.

  • use.latest.version: applies only when auto.register.schemas is false. When use.latest.version is true, the serializer uses the latest version of the schema in the subject for serialization and doesn’t derive a schema for the object passed to the client. For deserialization, when use.latest.version is true, the deserializer uses the latest version of the schema from Schema Registry to decode future messages. This setting doesn’t override the schema ID embedded in existing messages. The deserializer still tries to fetch the schema for the message’s schema ID, as shown in Wire format. If that schema ID is missing, such as after deletion, deserialization fails with a schema not found error.

  • latest.compatibility.strict: defaults to true and applies only when use.latest.version is true. When both properties are true, the serializer checks during serialization that the latest subject version is backward compatible with the schema of the object being serialized. If the check fails, the serializer throws an error. When latest.compatibility.strict is false, the serializer uses the latest subject version without a compatibility check. Setting latest.compatibility.strict to false can be useful when you use schema references.

The following table summarizes serializer behavior for these three properties.

auto.register.schemas

use.latest.version

latest.compatibility.strict

Behavior

true

true or false

true or false

The serializer attempts to register the schema with Schema Registry by deriving a schema for the object passed to the client. When auto.register.schemas is true, it overrides use.latest.version and latest.compatibility.strict. Their settings don’t matter.

false

true

false

The serializer uses the latest version of the schema in the subject for serialization.

false

true

true

The serializer checks that the latest subject version is backward compatible with the schema of the object being serialized. If the check fails, the serializer throws an error.

Turn off schema auto-registration and turn on use.latest.version in the following scenarios:

  • Combine multiple events in the same topic with schema references. You can use Schema references to combine multiple events in the same topic. For Avro and JSON Schema serializers, this configuration requires turning off schema auto-registration. For examples of configuring serializers to use the latest schema version without auto-registering schemas, see combining multiple event types in the same topic (Avro) and combining multiple event types in the same topic (JSON).

  • Improve production efficiency by avoiding Schema not found exceptions. Subtle differences that aren’t semantically significant can exist between a preregistered schema and the schema a client uses when the client uses code-generated classes from the preregistered schema with a Schema Registry-aware serializer. For example, with Protobuf, a fully qualified type name such as google.protobuf.Timestamp can generate a descriptor with the type name .google.protobuf.Timestamp. Schema Registry considers these two variations of the same type name to be different. With auto-registration on, this difference results in two essentially identical registered schemas. With auto-registration off, it can cause a Schema not found exception. To configure the serializer to not register new schemas and to ignore minor differences between client and registered schemas, set these properties in your serializer configuration:

    auto.register.schemas=false
use.latest.version=true
latest.compatibility.strict=false

    The use.latest.version property sets the serializer to retrieve the latest schema version for the subject and use it for validation and serialization, ignoring the client’s schema. This configuration assumes that any differences between the client schema and the latest registered schema are minor and backward compatible.

Specify schema ID and compatibility checks

You can also configure the following properties on Schema Registry clients:

  • use.schema.id: specifies the schema ID to use for serialization. By default, Schema Registry auto-assigns a numeric schema ID.

  • id.compatibility.strict: when true, which is the default, Schema Registry checks for backward compatibility between the schema with the given ID and the schema of the object to serialize.

Compatibility checks

All schema formats support the following compatibility levels, as described in Schema Evolution and Compatibility for Schema Registry on Confluent Platform:

  • BACKWARD

  • FORWARD

  • FULL

  • BACKWARD_TRANSITIVE

  • FORWARD_TRANSITIVE

  • FULL_TRANSITIVE

  • NONE

The ParsedSchema interface includes the method isBackwardCompatible(ParsedSchema previousSchema). When a schema type uses this method to define backward compatibility, Schema Registry can derive the other compatibility types from it.

The Avro specification defines the rules for Avro under Schema Resolution.

The rules for Protobuf backward compatibility come from the Protobuf language specification:

  • Fields can be added. All Protobuf fields are optional by default. Schema Registry uses the defaults you specify for backward compatibility.

  • Fields can be removed. A new field can reuse a field number if it has the same type. A new field can’t reuse a field number if it has a different type.

  • These groups of types are compatible, and you can swap the types within a group in the same field:

    • int32, uint32, int64, uint64, and bool

    • sint32 and sint64

    • string and bytes

    • fixed32 and sfixed32

    • fixed64 and sfixed64

  • enum is compatible with int32, uint32, int64, and uint64, and you can swap them in the same field.

  • Changing a single value into a member of a new oneof is compatible.

  • For string, bytes, and message fields, singular fields are compatible with repeated fields. Given serialized data of a repeated field as input, clients that expect a singular field take the last input value for a primitive type field or merge all input elements for a message type field. This behavior isn’t generally safe for numeric types, including bool and enum. Repeated fields of numeric types can use the packed format, which clients don’t parse correctly when they expect a singular field. For more information, see Updating a Message Type in the Protobuf documentation.

The rules for JSON Schema backward compatibility are more involved. For the rules, see JSON Schema compatibility rules.

Schema normalization

Schema normalization makes Schema Registry treat semantically equivalent schemas as the same schema. It’s off by default. As a best practice, turn on schema normalization per subject or globally.

When you register a schema or look up a schema ID, Schema Registry uses the string representation of the schema. Schema Registry performs minor formatting of the string representation but otherwise leaves the schema as is. As a result, Schema Registry can treat two semantically equivalent schemas as different.

To treat semantically equivalent schemas as the same schema, ask Schema Registry to normalize the schema during registration or lookup. Pass the configuration parameter normalize.schemas=true to the serializer, or pass the query parameter normalize=true to the REST APIs for registration and lookup. For more information, see POST /subjects/(string: subject)/versions and POST /subjects/(string: subject) in the Confluent Platform API reference, or Register a schema under a subject and Lookup schema under subject in the Confluent Cloud API reference.

Normalization handles syntactic differences such as:

  • The ordering of properties in JSON Schema

  • The ordering of imports and options in Protobuf

  • The ordering of schema references

  • Non-qualified names versus fully qualified names

You can also turn on schema normalization globally with the /config endpoint for Schema Registry, as described in the Confluent Platform API Reference and the Confluent Cloud API Reference.

Avro normalization

Schema Registry applies the following transformations to an Avro schema to produce its Parsing Canonical Form. These transformations resemble the Apache Avro® Parsing Canonical Form transformations in the specification under Transforming into Parsing Canonical Form, except for the STRIP transformation. Schema Registry SerDes exclude STRIP because it removes data from the docs, default, and alias fields.

  • Convert primitive schemas to their simple form, such as int for {"type":"int"}.

  • Replace short names with full names by using the applicable namespaces. Then remove the namespace attributes, which are redundant.

  • Order the fields of JSON objects as follows: name, type, fields, symbols, items, values, size. For example, if an object has type, name, and size fields, the name field comes first, followed by the type field and then the size field.

  • Replace escaped characters in all JSON string literals in the schema text, such as \uXXXX escapes, with their UTF-8 equivalents.

  • Remove the quotes and any leading zeros from JSON integer literals, which appear in the size attributes of fixed schemas.

  • Remove extra whitespace in JSON outside of string literals.

Kafka producers and consumers for development and testing

The Confluent and open source Apache Kafka® scripts for basic actions on Kafka clusters and topics are in $CONFLUENT_HOME/etc/bin. For a full reference of the Confluent premium command line tools and utilities, see CLI Tools for Confluent Platform. The tools include Confluent-provided producers and consumers that you can run locally against a self-managed Confluent Platform instance, the Confluent Platform demo, or Confluent Cloud clusters. The $CONFLUENT_HOME/etc/bin directory contains these producers and consumers:

  • kafka-avro-console-consumer

  • kafka-avro-console-producer

  • kafka-protobuf-console-consumer

  • kafka-protobuf-console-producer

  • kafka-json-schema-console-consumer

  • kafka-json-schema-console-producer

The same directory contains the original, generic kafka-console-consumer and kafka-console-producer, which expect an Avro schema by default. For a reference of the open source utilities, see Kafka Command-Line Interface (CLI) Tools.

Command-line producer, consumer utilities and JSON encoding of messages

Avro and Protobuf let you encode messages in either human-readable JSON or a storage-efficient binary format, as described in the respective specifications:

The command line utilities, REST Proxy, and Confluent Control Center use these JSON encodings.

To start the Protobuf command line producer, run:

kafka-protobuf-console-producer --bootstrap-server localhost:9092 --topic t1 --property value.schema='message Foo { required string f1 = 1; }'

To start the Protobuf command line consumer in a separate terminal, run:

kafka-protobuf-console-consumer --topic t1 --bootstrap-server localhost:9092

Send JSON messages in the form { "f1": "some-value" }.

To start the JSON Schema command line producer, run:

kafka-json-schema-console-producer --bootstrap-server localhost:9092 --topic t2 --property value.schema='{"type":"object","properties":{"f1":{"type":"string"}}}'

To start the JSON Schema command line consumer, run:

kafka-json-schema-console-consumer --topic t2 --bootstrap-server localhost:9092

Send JSON messages in the form { "f1": "some-value" }.

To pass references to the producers, use <key.refs> or <value.refs>. For example:

--property value.refs='[ { "name": "myName", "subject": "mySubject", "version": 1 } ]'

For more examples of these command line utilities, see the test drive for each format:

Troubleshoot Avro producers on Confluent Platform 7.5.2 and 7.4.3

This issue affects only the named versions of Confluent Platform. It doesn’t affect Confluent Cloud.

In Confluent Platform 7.5.2 and 7.4.3, io.confluent.kafka-avro-serialize can fail to serialize a schema that uses some logicalType values, such as timestamp-millis. When you produce Avro records with a schema that uses a logicalType, one or more of the following exceptions can occur, with both the REST Proxy and kafka-avro-console-producer.

The following schema field uses a logicalType:

{
  "name": "createdDate",
  "type": {
    "logicalType": "timestamp-millis",
    "type": "long"
  }
}

The producer can return one of the following errors:

error serializing Avro message

java.lang.ClassCastException: class java.time.Instant cannot be cast to class java.lang.Number
Caused by: org.apache.avro.AvroRuntimeException: Unknown datum type java.time.Instant

The REST Proxy can return the following error when you produce records:

{"error_code":40801,"message":"Error serializing Avro message"}

Confluent Platform 7.5.3 and 7.4.4 fix this issue, but you must add the following kafka-avro-console-producer property:

--property avro.use.logical.type.converters=true

REST Proxy versions earlier than 7.7.0 don’t support setting this property. As a workaround, use REST Proxy 7.4.2-2 or earlier, or 7.5.1-1 or earlier.

Basic authentication security for producers and consumers

Schema Registry can authenticate requests with Basic authentication headers. To send the headers, set the authentication properties in your producer or consumer configuration. For an example that uses a credentials file to authenticate a consumer to Schema Registry, see Print schema IDs with command line consumer utilities.

For more examples and details on credentials files on Confluent Cloud and Confluent Platform, see the Prerequisites section in the test drive for each schema format:

For all Schema Registry client configuration options, see Configuration Reference for Schema Registry Clients on Confluent Cloud and Configuration Reference for Schema Registry Clients on Confluent Platform.

Wire format

The wire format identifies the schema in either the payload prefix or the message header:

Wire format: schema ID in the payload prefix

In the payload prefix format, each message starts with a version byte, a 4-byte schema ID, and, for Protobuf, message indexes, followed by the data.

In most cases, use the serializers and formatter directly without worrying about how messages map to bytes. If you work with a language that Confluent hasn’t developed serializers for, or want to understand how Confluent Platform works, the following details describe how data maps to low-level bytes.

The wire format has these components:

Bytes

Area

Description

0

Version Byte

Confluent serialization format version number, which is 0 when using the schema ID, the default.

1-4

Schema ID

4-byte schema ID as returned by Schema Registry.

5-x

Messaging index

For Protobuf, an array of indexes that correspond to the message type. Otherwise, this is empty.

x+1…

Data

The data for the specified schema format, such as binary encoding for Avro or Protocol Buffers. The only exception is raw bytes, which Schema Registry writes directly without special encoding.

Important

  • The schema ID uses big-endian ordering, which is standard network byte order.

  • The wire format applies to Kafka message keys and message values.

The Protobuf serialization format appends a list of message indexes after the version byte and schema ID. The message indexes are an array of indexes that corresponds to the message type, which can be nested. A single Schema Registry Protobuf entry can contain multiple Protobuf messages, and those messages can contain nested messages. The message indexes identify which Protobuf message in the Schema Registry entry to use. For example, given a Schema Registry entry with the following definition:

package test.package;

message MessageA {
    message MessageB {
        message MessageC {
        ...
        }
    }
    message MessageD {
    ...
    }
    message MessageE {
        message MessageF {
        ...
        }
        message MessageG {
        ...
        }
    ...
    }
...
}
message MessageH {
    message MessageI {
    ...
    }
}

Reading the array backwards, the array [1, 0] is the first nested message type of the second top-level message type, which corresponds to test.package.MessageH.MessageI. Similarly, [0, 2, 1] is the second message type of the third message type of the first top-level message type, which corresponds to test.package.MessageA.MessageE.MessageG.

Schema Registry encodes the message indexes as int values with variable-length zigzag encoding, prefixed by the length of the array, which is also zigzag encoded. The message index encoding is the same as Avro, as described in Binary encoding in the Avro specification.

For example, Schema Registry encodes the array [1, 0] as the variable-length integers 2,1,0, where the first 2 is the length.

Schema Registry optimizes the most common case, the first message type, which the array [0] represents. The array would normally be encoded as 1,0, where 1 is the length, but Schema Registry encodes it as a single 0.

Messages with null keys or values pass broker-side schema ID validation. For details, see:

Wire format: schema GUID in header

In the schema GUID in header format, the serializer places a 16-byte schema GUID in a message header, and the payload contains only the serialized data.

As of Confluent Platform 8.1.1, you can change the wire format so the serializer doesn’t write the schema ID in the payload prefix.

This wire format simplifies migration when a client that doesn’t use Schema Registry starts to use it. Such clients have only the serialized data in the message key or value. Adopting a format that has the schema ID and other schema metadata in the payload prefix would be a breaking change for them.

To avoid the breaking change, configure the serializer to place the schema metadata in the message header. The header contains a 16-byte schema GUID in place of the 4-byte schema ID.

The serializer writes the header value to the __key_schema_id header for message keys and the __value_schema_id header for message values. The message payload contains only the serialized data. The header value has the following layout:

Bytes

Area

Description

0

Version Byte

Confluent serialization format version number, which is 1 when using the schema GUID.

1-16

Schema GUID

16-byte schema GUID as returned by Schema Registry.

17-x

Message indexes

For Protobuf, an array of indexes that correspond to the message type. Otherwise, the field is empty.

Schema Registry generates the 16-byte GUID from a fingerprint of the schema, including its schema references, rules, and metadata. Two schemas that are exactly the same, including schema references, rules, and metadata, have the same fingerprint regardless of which Schema Registry they were registered with. The Schema Registry APIs show the GUID in the 8-4-4-4-12 hexadecimal format, such as xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, where each x represents four bits. To retrieve a schema by its fingerprint GUID, use the /schemas/guids/{guid} endpoint.

To turn on the schema GUID in the header for message keys, message values, or both, set the corresponding properties:

key.schema.id.serializer=io.confluent.kafka.serializers.schema.id.HeaderSchemaIdSerializer

value.schema.id.serializer=io.confluent.kafka.serializers.schema.id.HeaderSchemaIdSerializer

Important

Starting with Confluent Platform 8.1.1, Schema Registry deserializers look for the schema GUID in the header first. If the header doesn’t have the GUID, the deserializer looks for the schema ID in the payload prefix. Earlier versions look only in the payload prefix for the schema ID.

Migrate to the schema GUID in header

Follow the upgrade order for your starting point to maintain compatibility during rollout.

Migrate from custom serialized Avro, Protobuf, or JSON data without Schema Registry

Upgrade producers first and then consumers.

  1. Producers: Adopt Schema Registry serializers for your chosen format and add the header schema serializer setting. Older consumers that ignore headers continue to work while you roll out producers.

    // Producer (JSON Schema example)
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
        "io.confluent.kafka.serializers.JSONSchemaSerializer");
props.put("value.schema.id.serializer",
        "io.confluent.kafka.serializers.schema.id.HeaderSchemaIdSerializer");

  2. Consumers: Adopt the Schema Registry Avro, Protobuf, or JSON deserializer. Deserializers check the header first and then the payload for backward compatibility. Because producers already write the header, consumers find the schema in the headers after the upgrade.

Migrate from Schema Registry with the schema ID in the payload prefix

Upgrade consumers first and then producers.

  1. Consumers: Upgrade first so they can read either the header or the payload. Existing producers keep writing the ID in the payload prefix until you finish upgrading consumers.

  2. Producers: Switch to header mode by adding the header schema serializer. Keep your existing serializer class, schema compatibility, and subject naming.

    // Producer (Avro example)
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
        "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("value.schema.id.serializer",
        "io.confluent.kafka.serializers.schema.id.HeaderSchemaIdSerializer");
Rollout guidance

  • Use canary producers and consumers to validate header reads while legacy readers keep working. Readers without Schema Registry ignore the header, and consumers with Schema Registry fall back to the payload.

  • For Protobuf subjects with multiple message types, confirm the message index behavior across nested messages in integration tests to ensure correct deserialization after migration.

  • Plan a staggered deployment per application or team to limit the scope of any problems and simplify rollbacks. Reverting producers to the payload ID or reverting consumers to legacy behavior is non-breaking with the recommended upgrade order.

Known behaviors and notes

  • In Confluent Platform 8.1.1 and later, deserializers prefer the header GUID and fall back to the payload ID if the header is absent. This behavior is intentional and supports rolling upgrades.

  • Kafka Streams support: in Confluent Platform 8.3 (Kafka Streams 4.3), headers-aware state stores preserve record headers. In Confluent Platform 8.2 and earlier, Kafka Streams state stores drop headers. For configuration and limitations, see Limitations of schema GUID in header.

  • ksqlDB support: if your ksqlDB applications rely on legacy payload ID parsing semantics, verify compatibility before rollout. ksqlDB support might be limited with header-only identification.

End-to-end examples

The following Avro producer writes the schema GUID to the message header:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "<brokers>");
props.put("schema.registry.url", "<sr-url>");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
          "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
          "io.confluent.kafka.serializers.KafkaAvroSerializer");

// Put schema identifier in Kafka header (value side)
props.put("value.schema.id.serializer",
          "io.confluent.kafka.serializers.schema.id.HeaderSchemaIdSerializer");

The following consumer checks the header first and falls back to the payload prefix:

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "<brokers>");
props.put("schema.registry.url", "<sr-url>");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
          "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
          "io.confluent.kafka.serializers.KafkaAvroDeserializer");

// No special config needed for header-first; CP 8.1.1+ deserializers check header, then fall back to the legacy payload prefix automatically.

Limitations of schema GUID in header

  • In Kafka Streams, state stores in Confluent Platform 8.2 and earlier drop record headers, which causes deserialization by downstream applications to fail when you use header-based schema identification.

    Starting in Confluent Platform 8.3 (Kafka Streams 4.3), headers-aware state stores preserve record headers through state store operations. To use schema GUIDs in headers with Kafka Streams, set dsl.store.format=HEADERS in your Kafka Streams application configuration. This setting enables Kafka Streams applications to adopt Schema Registry without modifying existing data, because Kafka Streams stores schema identifiers in record headers, which are metadata, and not in the payload prefix.

    The following Kafka Streams DSL operations don’t support headers-aware state stores in Confluent Platform 8.3:

    • suppress()

    • Left and outer stream-stream joins

    • Versioned state stores

  • When you use ksqlDB, state stores might drop headers, which causes deserialization by downstream applications to fail.

  • When you use Kafka Connect, verify that the Connect configurations preserve the header and don’t transform it.

  • Changing from the ID in the payload prefix to the GUID in the header for a record key might change the partitioning for the same raw key, which might cause downstream queries to fail.

  • When you use Schema Linking, don’t use configurations that change the body of the schema, such as kekRenameFormat. Changing the body causes the GUID to differ between the source and destination Schema Registry clusters, which leads to deserialization errors.

Compatibility guarantees across Confluent Platform versions

Confluent Platform guarantees that the serialization format its serializers use is stable over major releases. Confluent makes no changes without advance warning. This stability is critical because the serialization format affects how keys map across partitions. Many applications depend on keys with the same logical format reaching the same physical partition. For these applications, the physical byte format of serialized data must not change unexpectedly. Even the smallest modification can route records with the same logical key to different partitions, because Kafka routes messages to partitions based on the hash of the key.

To prevent variation as serializers gain new formats, the serializers change output formats conservatively. To guarantee stability for clients, Confluent Platform and its serializers ensure the following:

  • The format, including the magic byte, doesn’t change without significant warning over multiple Confluent Platform major releases. The default can change infrequently to allow adoption of new features by default. These changes happen conservatively, with at least one major release between changes, during which user-facing warnings tell users about the transition. Significant, compatibility-affecting changes guarantee at least one major release of warning and two major releases before an incompatible change.

  • Within the version specified by the magic byte, the format never changes in a backward-incompatible way. Any changes are fully backward compatible, with documentation in the release notes and at least one version of warning if a change introduces a new serialization feature that requires additional downstream support.

  • Deserialization support lasts over multiple major releases. This support isn’t guaranteed indefinitely, but Confluent Platform supports deserializing earlier formats for as long as no notified reason for incompatibility exists.

For more information about compatibility or support, ask the community mailing list.

Next steps

For details about your schema format, see:

Related content