Schema Evolution and Compatibility for Schema Registry on Confluent Cloud
Ready to get started?
Sign up for Confluent Cloud, the fully managed cloud-native service for Apache Kafka® and get started for free using the Cloud quick start.
Download Confluent Platform, the self managed, enterprise-grade distribution of Apache Kafka and get started using the Confluent Platform quick start.
Schema evolution
Schema evolution is the practice of safely changing schemas over time while maintaining compatibility with existing producers and consumers. Schema Registry enforces compatibility by comparing new schema versions against previous versions using configurable compatibility types.
Compatibility types define which schema changes are allowed:
Backward - consumers using new schema can read data written with old schema (add optional fields, remove fields)
Forward - consumers using old schema can read data written with new schema (remove optional fields, add fields)
Full - both backward and forward compatible (add/remove optional fields only)
Transitive - compatibility checked against all previous versions, not just the last one
The default compatibility mode is BACKWARD. Each schema version gets a unique ID and an incremented version number. When a schema is updated, Schema Registry checks compatibility before accepting the new version.
To learn more about serdes for supported schema formats, see “Formats, Serializers, and Deserializers” in either the Confluent Cloud documentation or in the Confluent Platform documentation
Compatibility types
Summary
The following tables summarize the compatibility rules for the different
schema formats: Avro, Protobuf, and JSON Schema for a given
subject. The Confluent Schema Registry default compatibility type is BACKWARD. The
sections that follow the tables describe all the compatibility types in more
detail.
The allowed changes are dependent on how fields are originally defined. For example, as described under Backward compatibility, the ability to delete a field and keep the schema compatible requires that the field was either specified as optional or provided a default value in the original version. To learn more, see Configuration Options on connectors to self-managed Schema Registry that provide further control over compatibility requirements.
For JSON Schema, compatibility behavior depends on both the compatibility
policy (compatibilityPolicy: LENIENT or STRICT) and the content
model (additionalProperties: true for open, false for closed). The tables
below show these in separate sections. A third policy, LOGICAL, applies across Avro, Protobuf, and JSON
Schema.
Avro and Protobuf compatibility
Avro |
Protobuf |
|||||
|---|---|---|---|---|---|---|
Allowed changes |
BW |
FW |
Full |
BW |
FW |
Full |
Add optional field |
✔ |
✔ |
✔ |
✔ |
✔ |
✔ |
Remove optional field |
✔ |
✔ |
✔ |
✔ |
✔ |
✔ |
Add required field |
✔ |
|||||
Remove required field |
✔ |
|||||
Add union/oneof variant |
✔ |
✔ |
||||
Remove union/oneof variant |
✔ |
✔ |
||||
Widen a scalar type |
✔ |
✔ |
✔ |
✔ |
||
Narrow a scalar type |
✔ |
✔ |
✔ |
✔ |
||
JSON Schema with lenient policy
Lenient policy allows schema changes regardless of the content model.
Allowed changes |
BW |
FW |
Full |
|---|---|---|---|
Add optional field |
✔ |
✔ |
✔ |
Remove optional field |
✔ |
✔ |
✔ |
Add required field |
✔ |
||
Remove required field |
✔ |
||
Add union/oneof variant |
✔ |
||
Remove union/oneof variant |
✔ |
||
Widen a scalar type |
✔ |
||
Narrow a scalar type |
✔ |
JSON Schema with strict policy
Strict policy compatibility rules vary by content model. Open content models
(additionalProperties: true) allow more changes than closed models
(additionalProperties: false).
Open content model (additionalProperties: true or omitted):
Allowed changes |
BW |
FW |
Full |
|---|---|---|---|
Add optional field |
✔ |
||
Remove optional field |
✔ |
||
Add required field |
✔ |
||
Remove required field |
✔ |
||
Add union/oneof variant |
✔ |
||
Remove union/oneof variant |
✔ |
||
Widen a scalar type |
✔ |
||
Narrow a scalar type |
✔ |
Closed content model (additionalProperties: false):
Allowed changes |
BW |
FW |
Full |
|---|---|---|---|
Add optional field |
✔ |
||
Remove optional field |
✔ |
||
Add required field |
|||
Remove required field |
|||
Add union/oneof variant |
✔ |
||
Remove union/oneof variant |
✔ |
||
Widen a scalar type |
✔ |
||
Narrow a scalar type |
✔ |
Table Legend:
✔ = Compatible operation for the given schema format and compatibility type
Blank space = Incompatible operation for the given schema format and compatibility type
BW = Backward compatible
FW = Forward compatible
Full = Fully compatible (both backward and forward)
Note
A REST API call to compatibility mode is global, meaning it overrides any compatibility parameters set in Schema Registry properties files. This is discussed in Using compatibility types below and shown in the API usage examples in Confluent Cloud: Schema Compatibility (V1) API Usage Examples and on Confluent Platform: Update compatibility requirements globally.
Schema Registry supports a global configuration context using the special context name
:.__GLOBAL:. When looking up compatibility settings withdefaultToGlobal, the lookup proceeds from the subject to the context, and then to the global context. For contexts without global configuration, compatibility is handled per context. To learn more about contexts, see Contexts.
Logical compatibility policy
The LOGICAL compatibility policy allows only the schema changes that
Flink, Tableflow, and Lightning Tables all support. Use it on subjects that
feed SQL queries or tables. The policy prevents a
schema change from suspending a Tableflow table, failing a Flink statement,
or causing existing records to be misread.
Tip
If a schema registration failed with a LOGICAL error, see
Read and fix a LOGICAL error.
Schema Registry offers three compatibility policies: STRICT (the default),
LENIENT, and LOGICAL.
STRICTandLENIENTcheck that Kafka consumers using Avro, Protobuf, or JSON Schema deserializers can read records written with a new schema version. The two differ only for JSON Schema. To learn more, see Compatibility types.LOGICALadds checks based on how Flink maps the schema to SQL types and how Tableflow and Lightning Tables map it to Apache Iceberg™ table types. Some changes pass the format’s own rules but still break these engines. For example,BACKWARDallows deleting a field. Tableflow doesn’t support dropping columns and suspends materialization for the topic. For the schema changes that Tableflow supports, see Schema compatibility and evolution.
The policy works alongside the subject’s compatibility type, such as
BACKWARD. The compatibility type selects the earlier versions that Schema Registry
compares a new schema with. The policy decides which rules the comparison uses.
LOGICAL applies to every subject it covers, even if Flink, Tableflow,
or Lightning Tables doesn’t currently read that subject. If you need a
change that LOGICAL blocks, you can override the policy for one
registration with ?force=true. See Force a change that LOGICAL blocks.
How LOGICAL checks a schema
When you register a schema under LOGICAL, Schema Registry runs three kinds of
checks. The registration fails if any of them fails.
Format compatibility: the native Avro, Protobuf, or JSON Schema check for the subject’s compatibility type. For JSON Schema, this check uses
LENIENTrules, becauseLOGICALadds its own checks for the properties that become columns. Avro and Protobuf checks are the same under every policy.Change checks: compare the new schema with earlier versions and reject changes that break Flink, Tableflow, or Lightning Tables. See Changes that LOGICAL blocks.
Validity checks: confirm that the schema itself maps to a SQL type and a table. These run on every registration, including the first version of a subject. See Schemas that LOGICAL rejects on every registration.
Before you begin
LOGICALworks only with theBACKWARD,BACKWARD_TRANSITIVE,FULL, andFULL_TRANSITIVEcompatibility types. Tableflow and Lightning Tables support only backward-compatible evolution. Schema Registry rejectsLOGICALwithNONE,FORWARD, orFORWARD_TRANSITIVE.Turning on
LOGICALdoesn’t re-check schemas that are already registered. Schema Registry checks new schemas against earlier versions, and the current schema must pass the validity checks. If the current schema contains something the validity checks reject, such as a self-referencing type, or the subject’s history contains a change thatLOGICALblocks, later registrations fail. Roll out LOGICAL shows how to check both.While a subject uses
LOGICAL, you can’t permanently delete the subject or its versions (error code42205). To permanently delete them, first turn offLOGICALfor the subject.For tables, the checks use the Iceberg type system. A Tableflow table backed by Delta Lake isn’t checked against Delta Lake’s type system.
Set the policy on the subject that holds the schema. If a topic’s subject is an alias, set it on the subject the alias points to. Registrations through the alias then follow that policy. A policy set on the alias itself has no effect, even though the alias’s configuration shows it.
Roll out LOGICAL
Turn on LOGICAL one subject at a time, starting with the subjects that
feed Flink, Tableflow, or Lightning Tables. For subjects that feed
Tableflow, use BACKWARD_TRANSITIVE. Each new schema is then checked
against every earlier version that the table already holds.
The API examples use the SR_APIKEY, SR_APISECRET, and
SCHEMA_REGISTRY_ENDPOINT shell variables, and jq. To find the Schema Registry values, see
Setup and suggestions.
export SR_APIKEY=<sr_api_key>
export SR_APISECRET=<sr_api_secret>
export SCHEMA_REGISTRY_ENDPOINT=<sr_endpoint_url>
Record the subject’s configuration to restore it later. Run the request twice: without
defaultToGlobalto see what the subject sets itself, and withdefaultToGlobal=trueto see the settings in effect.curl --silent -u $SR_APIKEY:$SR_APISECRET \ "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>" curl --silent -u $SR_APIKEY:$SR_APISECRET \ "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>?defaultToGlobal=true"
If the effective
compatibilityLevelisn’tBACKWARD,BACKWARD_TRANSITIVE,FULL, orFULL_TRANSITIVE, or the subject feeds Tableflow and isn’tBACKWARD_TRANSITIVE, set it. For example:curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \ "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
Test your current schema against the validity checks. Give an empty scratch subject the
LOGICALpolicy, and check the schema against it. The compatibility endpoint runs the validity checks even when a subject has no versions, and it doesn’t register anything:# Copy the latest schema, with its type and references curl --silent -u $SR_APIKEY:$SR_APISECRET \ "$SCHEMA_REGISTRY_ENDPOINT/subjects/<subject_name>/versions/latest" \ | jq '{schema, schemaType, references}' > scratch.json # Give the scratch subject the LOGICAL policy curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"compatibility": "BACKWARD", "compatibilityPolicy": "LOGICAL"}' \ "$SCHEMA_REGISTRY_ENDPOINT/config/logical-test" # Check the schema against it curl --silent -u $SR_APIKEY:$SR_APISECRET -X POST \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data @scratch.json \ "$SCHEMA_REGISTRY_ENDPOINT/compatibility/subjects/logical-test/versions?verbose=true" # Remove the scratch configuration curl --silent -u $SR_APIKEY:$SR_APISECRET -X DELETE \ "$SCHEMA_REGISTRY_ENDPOINT/config/logical-test"
If
is_compatibleisfalse, fix the schema before you continue. See Schemas that LOGICAL rejects on every registration.Turn on
LOGICAL, in the Cloud Console or with the API.In the Cloud Console, open the subject, click the edit icon next to Compatibility policy, select Logical, and click Save.
With the API, send this request:
curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"compatibilityPolicy": "LOGICAL"}' \ "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
If the compatibility type isn’t supported, the request fails with error code
42203:compatibilityLevel='NONE' doesn't work with compatibilityPolicy='LOGICAL'. Use BACKWARD, BACKWARD_TRANSITIVE, FULL, or FULL_TRANSITIVE.
Run the second request in step 1 again, and confirm that
compatibilityPolicyisLOGICAL.Check the latest schema against the subject, with the request in Check schemas in CI. Under
BACKWARD_TRANSITIVEandFULL_TRANSITIVE, this compares it with every earlier version. If a change check fails, the subject’s history already contains a change thatLOGICALblocks, and every later registration fails the same way. UseBACKWARDinstead, which compares only with the latest version, or turn offLOGICALfor the subject.Add a
LOGICALcheck to your CI pipeline. See Check schemas in CI.
Set LOGICAL for the whole registry
To make LOGICAL the default for every subject that doesn’t set its own
policy, send the same request to /config:
curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \
-H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"compatibilityPolicy": "LOGICAL"}' \
"$SCHEMA_REGISTRY_ENDPOINT/config"
Warning
Every subject that inherits the setting is then checked, including
subjects that no SQL engine reads. Registrations fail anywhere a schema
can’t be represented as a table. Do this only after you roll out
LOGICAL on individual subjects.
Check schemas in CI
The compatibility endpoint runs the same checks as registration, without registering anything. Without a version number, it compares the schema with the versions that the subject’s compatibility type selects, the same ones registration uses.
Put the request body in a file, for example schema.json. Let jq
escape the schema for you:
# Avro
jq -n --rawfile s order.avsc '{schema: $s}' > schema.json
# Protobuf
jq -n --rawfile s order.proto '{schemaType: "PROTOBUF", schema: $s}' > schema.json
# JSON Schema
jq -n --rawfile s order.json '{schemaType: "JSON", schema: $s}' > schema.json
If the schema uses references, add them to the body in references, as
you do when you register it.
The endpoint returns HTTP 200 for compatible and incompatible schemas.
Fail the build on the is_compatible field. Other status codes, such as
422 for an unparsable schema, mean the check didn’t run:
status=$(curl --silent -o result.json -w '%{http_code}' \
-u $SR_APIKEY:$SR_APISECRET -X POST \
-H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data @schema.json \
"$SCHEMA_REGISTRY_ENDPOINT/compatibility/subjects/<subject_name>/versions?verbose=true")
if [ "$status" != "200" ]; then cat result.json; exit 1; fi
jq -e '.is_compatible' result.json > /dev/null \
|| { jq -r '.messages[]' result.json; exit 1; }
When the check fails, messages lists each problem in the same form as a
registration error. See Read and fix a LOGICAL error.
A change you plan to force fails this check too. Run the ?force=true
registration as a separate step that someone reviews and approves.
Changes that LOGICAL allows
Under BACKWARD and BACKWARD_TRANSITIVE, you can evolve a schema in
these ways:
Add an optional field.
Make a
NOT NULLfield nullable. In Protobuf, mark itoptional. In JSON Schema, remove it fromrequired.Widen an integer, for example
INTtoBIGINT(Avrointtolong, Protobufint32toint64). JSON Schemaintegerhas no width. This change doesn’t apply to it.Increase a decimal’s precision while keeping its scale.
Increase the length of a
VARCHARorVARBINARY.Add an enum symbol.
Under FULL and FULL_TRANSITIVE, LOGICAL also checks each change
in reverse. It rejects widening a type, increasing a decimal’s precision or a
length, making a field nullable, and adding an enum symbol. Adding an optional
field is the only change in the preceding list that FULL allows.
Changes that LOGICAL blocks
The tables in this section list each change that LOGICAL rejects, where
it breaks, how to make the change safely, and the rule name that appears as
errorType in the error.
Breaks lists the engines that the change affects when it reaches them: Tableflow suspends materialization for the topic, a Flink statement fails, or, for type changes, the engine reads existing values incorrectly with no error. Any query or job that references a changed field still needs updating.
The examples use Logical Type (SQL) names. To see how each type maps to Avro, Protobuf, and JSON Schema, see Logical Type mappings. For what nullable and optional mean in each format, see Nullable and optional fields in each format.
Fields
Change |
What goes wrong |
Breaks |
How to fix |
Rule |
|---|---|---|---|---|
Delete a field, including a field with a default value |
Tableflow doesn’t support dropping columns. It suspends materialization for the topic. |
Tableflow, Lightning Tables |
Keep the field, and retire it instead. See Retire a field. |
|
Rename a field |
Renaming a field isn’t supported, including with Avro |
Tableflow, Lightning Tables |
Add the new name as an optional field, then retire the old one. See Rename a field. |
|
Add a required field, meaning any field that isn’t nullable, even if it has a default value |
Existing records have no value for the field. |
Tableflow, Lightning Tables |
Add the field as optional. See Nullable and optional fields in each format. |
|
Note
LOGICAL doesn’t check field order in Confluent Cloud. A Tableflow table
can’t apply a new column order. Keep the original order and add new
fields at the end.
Types
Change |
What goes wrong |
Breaks |
How to fix |
Rule |
|---|---|---|---|---|
Change a field to an unrelated type |
A string field becomes an integer. |
Flink, Tableflow, Lightning Tables |
Keep the type. Add a new field with the new type, and retire the old one. |
|
Narrow a type so it can’t hold all existing values |
Protobuf |
Flink, Tableflow, Lightning Tables |
Keep the wider type, and check the range in the producer. |
|
Promote an integer to a floating-point type |
Avro |
Flink, Tableflow, Lightning Tables |
Keep the integer type, or add a new floating-point field. |
|
Turn a date into a timestamp |
Avro |
Flink, Tableflow, Lightning Tables |
Keep the date field, and add a new timestamp field. |
|
Change time or timestamp precision |
Avro |
Flink |
Keep the original precision. |
|
Change a decimal’s scale, or reduce its precision |
|
Flink, Tableflow, Lightning Tables |
Keep the scale. Under |
|
Shorten a |
|
Flink |
Keep the original length. Under |
|
Change a fixed length |
|
Flink, Tableflow, Lightning Tables |
Keep the original length. |
|
Change a map’s key type |
|
Flink, Tableflow, Lightning Tables |
Keep the key type. Add a new map field if you need a different key. |
|
Nullability, defaults, and enums
Change |
What goes wrong |
Breaks |
How to fix |
Rule |
|---|---|---|---|---|
Make a nullable field |
Existing records can contain null values. |
Flink, Tableflow, Lightning Tables |
Keep the field nullable, and require a value in the producer instead. |
|
Remove the default value from a |
Flink uses the default when it reads records that don’t contain the field. |
Flink |
Restore the default value. |
|
Remove an enum symbol |
Existing records can use the symbol. |
Tableflow, Lightning Tables |
Keep every existing symbol, and stop producing the one you want to retire. |
|
Nullable and optional fields in each format
Format |
A field is nullable when |
Add an optional field |
|---|---|---|
Avro |
Its type is a union with |
|
Protobuf |
It’s marked |
Add a field with a new tag number, marked |
JSON Schema |
It isn’t listed in |
Add the property without listing it in |
Logical Type (SQL), see Logical Type mappings |
It’s declared without |
Declare the field without |
Retire a field
Because LOGICAL blocks deleting a field, retire it instead:
If the field is
NOT NULL, make it nullable.BACKWARDandBACKWARD_TRANSITIVEallow this change.FULLandFULL_TRANSITIVEdon’t allow it. Under those types, change the subject toBACKWARD_TRANSITIVEfirst.Update producers to stop writing the field. New records then have no value for it.
Update Flink statements and other consumers to stop reading the field.
The column stays in the table and holds null for new records.
Rename a field
Add the new name as an optional field.
Update producers to write both fields.
Move Flink statements and other consumers to the new field.
Retire the old field.
Records written before the change have null in the new column, and their
values stay in the old column. Until you backfill, read both, for example
COALESCE(new_name, old_name).
Schemas that LOGICAL rejects on every registration
These checks look at the schema by itself. If a registered schema fails one
of them, fixing it can require a change that LOGICAL blocks.
Register the fix with ?force=true. Under transitive compatibility types,
later registrations can need it too. See Force a change that LOGICAL blocks.
Schema contains |
Limit |
Breaks |
Rule |
|---|---|---|---|
A type that refers to itself |
Not supported |
Flink, Tableflow, Lightning Tables |
|
A reference to a type that the schema doesn’t define |
Not supported |
Flink, Tableflow, Lightning Tables |
|
A decimal precision or scale outside the supported range |
Precision 1 to 38, scale 0 to the precision |
Flink, Tableflow, Lightning Tables |
|
A |
At least 1 |
Flink, Tableflow, Lightning Tables |
|
Nesting deeper than the supported number of levels |
100 levels |
Flink, Tableflow, Lightning Tables |
|
A record, message, or union with no fields or types |
At least one |
Tableflow, Lightning Tables |
|
Two field names that differ only by letter case, such as |
Not supported |
Tableflow, Lightning Tables |
|
A |
Not supported |
Tableflow, Lightning Tables |
|
A timestamp with precision greater than 6 |
6 (microseconds) |
Tableflow, Lightning Tables |
|
Read and fix a LOGICAL error
When a registration fails, Schema Registry returns HTTP 409 with error_code
40901 and a message. The message lists each problem with its rule in
errorType and a description that names the field. For example,
deleting the carrier field from a BACKWARD subject returns:
{
"error_code": 40901,
"message": "Schema being registered is incompatible with an earlier schema for subject \"shipment-orders\", details: [{errorType:\"FIELD_DELETED\", category:[\"ICEBERG_V2\"], description:\"Schema no longer includes field 'carrier'. Restore the field.\", additionalInfo:\"carrier\"}, {validateFields: \"true\", compatibility: \"BACKWARD\"}]"
}
The Cloud Console shows the same message below the schema editor. The
entries inside details aren’t strict JSON. To detect a rule in a
script, match on the text errorType:"<RULE>". Entries from LOGICAL
include a category that names the engines the change breaks
(FLINK, ICEBERG_V2, or both). Entries without a category come
from the format’s own compatibility check.
Find the errorType in the following table. Rules not listed here come
from the format’s own compatibility check, which ?force=true can’t
bypass.
|
Breaks |
How to fix |
Details |
|---|---|---|---|
|
Flink, Tableflow, Lightning Tables ( |
Change the schema so it stays within the supported limits. If a
registered version already fails the check, register the fix with
|
|
|
Tableflow, Lightning Tables |
Add a field, or rename one of the colliding fields. If a registered
version already fails the check, register the fix with
|
|
|
Tableflow, Lightning Tables |
Keep every existing symbol. |
|
|
Tableflow, Lightning Tables |
Add the field back, then retire or rename it. |
|
|
Tableflow, Lightning Tables (Flink reports
|
Keep the map’s key type. |
|
|
Flink |
Restore the default value. |
|
|
Flink, Tableflow, Lightning Tables |
Keep the field nullable. |
|
|
Tableflow, Lightning Tables |
Add the field as optional. |
|
|
Depends on the change. See the Breaks column in Changes that LOGICAL blocks. |
Keep the original type, or add a new field with the new type. |
Force a change that LOGICAL blocks
If you need a change that LOGICAL blocks, and you accept that it can
break Flink, Tableflow, or Lightning Tables, add ?force=true to the
registration request:
curl --silent -u $SR_APIKEY:$SR_APISECRET -X POST \
-H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data @schema.json \
"$SCHEMA_REGISTRY_ENDPOINT/subjects/<subject_name>/versions?force=true"
Anyone who can register schemas in the subject can use force. They don’t
need another role, and you can’t turn force off for a subject or the
registry.
force skips all LOGICAL checks, both the change checks and the
validity checks, for that one registration. It doesn’t skip the format
compatibility check. force can’t register a schema that the format’s own
rules reject.
To decide whether a change is safe to force, check its Breaks column in Changes that LOGICAL blocks or Schemas that LOGICAL rejects on every registration. For example, if only Flink reads a subject, a change that breaks only Tableflow and Lightning Tables doesn’t affect it today, but it does affect any table you create from the subject later.
Kafka consumers don’t get deserialization errors after a forced change, but they can read wrong values with no error. For example:
After a forced
int64toint32change, consumers using the new schema silently truncate large values.After a forced change that removes
optionalfrom a proto3 field, consumers using the new schema read an unset value as the type’s default (0,"", orfalse). They don’t read it as null.
Before you force a change:
Plan for the engines it breaks. For example, after you force a field deletion, Tableflow suspends materialization for the topic, because it doesn’t support dropping columns. Update the Flink statements that read the subject. See Schema compatibility and evolution.
Under
BACKWARD_TRANSITIVEandFULL_TRANSITIVE, every later version is also compared with the versions before the forced one. A forced change such as a field deletion then fails every later registration. Each one also needs?force=true, which skips allLOGICALchecks for it. Your CI check also fails for each one.
Turn off LOGICAL
Restore what you recorded in step 1 of Roll out LOGICAL:
If the subject set its own policy, set
compatibilityPolicyback to that value.STRICTis the default.curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"compatibilityPolicy": "STRICT"}' \ "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
If the subject inherited its policy from the registry, remove the subject’s own policy. The subject keeps its compatibility type and inherits the registry’s policy again:
curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"compatibilityPolicy": null}' \ "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
Don’t use
DELETE /config/<subject_name>for this. It also removes the subject’s compatibility type.If you changed the compatibility type in step 2, set
compatibilityback to the value you recorded.To turn off a registry-wide
LOGICALsetting, send the first request in this list to/config.
In the Cloud Console, select Strict or Lenient in the
Compatibility policy dialog. If you added a LOGICAL check to your CI
pipeline, remove it.
Avro, Protobuf, and JSON Schema have different compatibility rules
Schema evolution and compatibility rules vary somewhat based on schema format. The scenarios and source code examples given on this page are geared for Avro, which was the first serializer and deserializer that Confluent supported. Avro was developed with schema evolution in mind, and its specification clearly states the rules for backward compatibility. The rules and grammar for JSON Schema and Protobuf can be more nuanced.
Although the general concepts apply to all formats, the details on how compatibility is implemented differs depending on whether you use Avro, JSON Schema, or Protobuf. For more in-depth explanations and specific examples on the newer formats, see the following blog posts and documentation:
Protobuf
Blog post: Understanding Protobuf Compatibility
Confluent Cloud documentation: Protobuf compatibility rules
Confluent Platform documentation: Protobuf compatibility rules
Note that best practice for Protobuf is to use BACKWARD_TRANSITIVE, as adding new message types is not forward compatible.
JSON Schema
Blog post: Understanding JSON Schema Compatibility
Confluent Cloud documentation: JSON Schema compatibility rules
Confluent Platform documentation: JSON Schema compatibility rules
Confluent Platform documentation: Compatibility checks
Backward compatibility
BACKWARD compatibility means that consumers using the new schema can read data produced with the last schema.
For example, if there are three schemas for a subject that change in order X-2, X-1, and X then BACKWARD compatibility ensures that consumers using the new schema X can process data written by producers using schema X or X-1, but not necessarily X-2.
If the consumer using the new schema needs to be able to process data written by all registered schemas, not just the last two schemas, then use BACKWARD_TRANSITIVE instead of BACKWARD.
For example, if there are three schemas for a subject that change in order X-2, X-1, and X then BACKWARD_TRANSITIVE compatibility ensures that consumers using the new schema X can process data written by producers using schema X, X-1, or X-2.
BACKWARD: consumer using schemaXcan process data produced with schemaXorX-1BACKWARD_TRANSITIVE: consumer using schemaXcan process data produced with schemaX,X-1, orX-2
Important
The Confluent Schema Registry default compatibility type is
BACKWARD, notBACKWARD_TRANSITIVE. The main reason thatBACKWARDcompatibility mode is the default, and preferred for Kafka, is so that you can rewind consumers to the beginning of the topic. WithFORWARDcompatibility mode, you aren’t guaranteed the ability to read old messages. Also,FORWARDcompatibility mode is harder to work with. In a sense, you need to anticipate all future changes. For example, inFORWARDcompatibility mode with Protobuf, you can’t add new message types to a schema.For Kafka Streams, only BACKWARD compatibility is supported. To learn more, see the note about Kafka Streams under Order of Upgrading Clients.
To understand how Avro uses a newer backward-compatible schema to decode (deserialize) data that was encoded (serialized) with an older schema, see ResolvingDecoder in the Apache Avro project.
An example of a backward compatible change is removing a field. A consumer developed to process events without this field can process events written with the old schema, which contain the field. The consumer ignores that field.
Consider the case where all of the data in Kafka is also loaded into HDFS, and you want to run SQL queries (for example, using Apache Hive) over all the data. Here, it is important that the same SQL queries continue to work even as the data is undergoing changes over time. To support this kind of use case, you can evolve the schemas in a backward compatible way. All Formats, Serializers, and Deserializers have rules as to what changes are allowed in the new schema for it to be backward compatible. For example, here are the Avro rules for compatibility. If all schemas are evolved in a backward compatible way, you can always use the latest schema to query all the data uniformly.
For example, an application can evolve the
user schema from the previous section to the following by adding a new field
favorite_color:
{"namespace": "example.avro",
"type": "record",
"name": "user",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": "int"},
{"name": "favorite_color", "type": "string", "default": "green"}
]
}
Note that the new field favorite_color has the default value “green”. This allows data encoded with the old schema
to be read with the new one. The default value specified in the new schema is used for the missing field when
deserializing the data encoded with the old schema. Had the default value been omitted in the new field, the new
schema would not be backward compatible with the old one since it’s not clear what value should be assigned to the new
field, which is missing in the old data.
Forward compatibility
FORWARD compatibility means that data produced with a new schema can be read by consumers using the last schema, even though they may not be able to use the full capabilities of the new schema.
For example, if there are three schemas for a subject that change in order X-2, X-1, and X then FORWARD compatibility ensures that data written by producers using the new schema X can be processed by consumers using schema X or X-1, but not necessarily X-2.
If data produced with a new schema needs to be read by consumers using all registered schemas, not just the last two schemas, then use FORWARD_TRANSITIVE instead of FORWARD.
For example, if there are three schemas for a subject that change in order X-2, X-1, and X then FORWARD_TRANSITIVE compatibility ensures that data written by producers using the new schema X can be processed by consumers using schema X, X-1, or X-2.
FORWARD: data produced using schemaXcan be read by consumers with schemaXorX-1FORWARD_TRANSITIVE: data produced using schemaXcan be read by consumers with schemaX,X-1, orX-2
An example of a forward compatible schema modification is adding a new field. In most data formats, consumers that were written to process events without the new field can continue doing so even when they receive new events that contain the new field.
Consider a use case where a consumer has application logic tied to a particular version of the schema. When the schema
evolves, the application logic may not be updated immediately. Therefore, you need to be able to project data with newer
schemas onto the (older) schema that the application understands. To support this use case, you can evolve the schemas
in a forward compatible way: data encoded with the new schema can be read with the old schema. For example, the new
user schema shown in the previous section on backward compatibility is also
forward compatible with the old one. When projecting data written with the new schema to the old one, the new field is
simply dropped. Had the new schema dropped the original field favorite_number (number, not color), it would not be
forward compatible with the original user schema since consumers wouldn’t know how to fill in the value for favorite_number
for the new data because the original schema did not specify a default value for that field.
Full compatibility
FULL compatibility means schemas are both backward and forward compatible.
Schemas evolve in a fully compatible way: old data can be read with the new schema, and new data can also be read with the last schema.
For example, if there are three schemas for a subject that change in order X-2, X-1, and X then FULL compatibility ensures that consumers using the new schema X can process data written by producers using schema X or X-1, but not necessarily X-2, and that data written by producers using the new schema X can be processed by consumers using schema X or X-1, but not necessarily X-2.
If the new schema needs to be forward and backward compatible with all registered schemas, not just the last two schemas, then use FULL_TRANSITIVE instead of FULL.
For example, if there are three schemas for a subject that change in order X-2, X-1, and X then FULL_TRANSITIVE compatibility ensures that consumers using the new schema X can process data written by producers using schema X, X-1, or X-2, and that data written by producers using the new schema X can be processed by consumers using schema X, X-1, or X-2.
FULL: backward and forward compatible between schemasXandX-1FULL_TRANSITIVE: backward and forward compatible between schemasX,X-1, andX-2
In Avro and Protobuf, you can define fields with default values. In that case, adding or removing a field with a default value is a fully compatible change.
In Avro, a field is made nullable (and therefore “optional” at the schema level) by using a union type that includes "null" (for example, ["null", "string"]).
Providing a default value is not required for nullability itself, but it is important for schema evolution when adding a new field so that existing data can still be read.
Avro requires that the default value conform to the first branch of the union, so it is common to put "null" first and use "default": null for fields that are intended
to be optional for schema evolution. Note that Avro does not have a built-in optional keyword like Protobuf; instead, the union-with-null pattern serves this purpose.
Compatibility rules for supported schema types are described in Compatibility checks.
JSON Schema does not explicitly define compatibility rules, so this blog post further explains how JSON Schema compatibility works, including full compatibility.
No compatibility checking
NONE compatibility type means schema compatibility checks are disabled.
Sometimes you need to make incompatible changes.
For example, modifying a field type from Number to String.
In this case, you need to either upgrade all producers and consumers to the new schema version at the same time or, more likely, create a brand-new topic and start migrating applications to use the new topic and new schema, avoiding the need to handle two incompatible versions in the same topic.
Transitive property
Transitive compatibility checking is important once you have more than two versions of a schema for a given subject. If compatibility is configured as transitive, then it checks compatibility of a new schema against all previously registered schemas; otherwise, it checks compatibility of a new schema only against the latest schema.
For example, if there are three schemas for a subject that change in order X-2, X-1, and X then:
transitive: ensures compatibility between X-2 <==> X-1 and X-1 <==> X and X-2 <==> X
non-transitive: ensures compatibility between X-2 <==> X-1 and X-1 <==> X, but not necessarily X-2 <==> X
Refer to an example of schema changes which are incrementally compatible, but not transitively so.
The Confluent Schema Registry default compatibility type BACKWARD is non-transitive, which means that it’s not BACKWARD_TRANSITIVE.
As a result, new schemas are checked for compatibility only against the latest schema.
Order of upgrading clients
The configured compatibility type has an implication on the order for upgrading client applications, that is, the producers using schemas to write events to Kafka and the consumers using schemas to read events from Kafka. Depending on the compatibility type:
BACKWARDorBACKWARD_TRANSITIVE: there is no assurance that consumers using older schemas can read data produced using the new schema. Therefore, upgrade all consumers before you start producing new events.FORWARDorFORWARD_TRANSITIVE: there is no assurance that consumers using the new schema can read data produced using older schemas. Therefore, first upgrade all producers to using the new schema and make sure the data already produced using the older schemas are not available to consumers, then upgrade the consumers.FULLorFULL_TRANSITIVE: there are assurances that consumers using older schemas can read data produced using the new schema and that consumers using the new schema can read data produced using older schemas. Therefore, you can upgrade the producers and consumers independently.NONE: compatibility checks are disabled. Therefore, you need to be cautious about when to upgrade clients.
Important
For Kafka Streams only FULL, TRANSITIVE, and BACKWARD compatibility is supported. For a plain consumer, it is safe to upgrade the consumer to the new schema after the producer is upgraded because a plain consumer reads only from the input topic. For Kafka Streams, the scenario is different. When you upgrade Kafka Streams, it also can read from the input topic (that now contains data with the new schema). However, in contrast to a plain consumer, Kafka Streams must also be able to read the old schema (from the state or changelog). Therefore, only
BACKWARDcompatibility is supported. The Kafka Streams apps must be upgraded first, then it’s safe to upgrade the upstream producer that writes into the input topic.FULL,FULL_TRANSITIVE, andBACKWARD_TRANSITIVEcompatibilities are always supported for Kafka Streams, as they include backward compatibility and so are, in effect, “stronger” settings thanBACKWARD.With Confluent Cloud for Apache Flink®,
FULL,FULL_TRANSITIVE, andBACKWARD_TRANSITIVEcompatibility types are supported. For more information, see Schema and Statement Evolution with Flink.
Specify schema compatibility requirements per subject
You can configure and update schema compatibility requirements globally or on a per-subject basis. To learn more, see Compatibility in the Schema Registry API Reference and the following topics in the Schema Registry API Usage Examples and Reference docs:
On Confluent Cloud:
On Confluent Platform:
Examples
Each of the sections above has an example of the compatibility type. An additional reference for Avro is Avro compatibility test suite, which presents multiple test cases with two schemas and the respective result of the compatibility test between them.
Using compatibility types
Compatibility rules and references for all supported schema types are described in Compatibility checks in Formats, Serializers, and Deserializers, and in Schema Evolution and Compatibility (on Confluent Cloud) and Schema Evolution and Compatibility (on Confluent Platform).
For details on how to use Schema Registry to store schemas and enforce compatibility rules during schema evolution, see the REST API usage examples and references for Confluent Cloud and Confluent Platform:
For Confluent Cloud, see Confluent Cloud Schema Registry REST API Usage Examples and Cloud REST API reference: Schemas (V1) and Subjects (v1)
For Confluent Platform, see Confluent Platform Schema Registry API Usage Examples and Schema Registry API Reference
Here are some tips to get you started.
To check the currently configured global compatibility type
On Confluent Cloud:
Use the REST API to Show global compatibility level currently in effect
Use the Confluent Cloud Console to View Global Compatibility Settings on Schemas
On Confluent Platform:
Use the REST API to Get the top level config.
To set the global or per schema compatibility level
You can configure schema compatibility in the following ways:
On Confluent Cloud:
Use the API to Update global compatibility level
Use the API to Update the subject compatibility level
Use the Confluent Cloud Console to View and Edit Global Compatibility Settings on Schemas
Use the Confluent Cloud Console as described in Changing subject level (per topic) compatibility mode of a schema
On Confluent Platform:
Update compatibility in your client application
Use the API to Update global compatibility
Use the API to Update subject compatibility
Use the Control Center (Legacy) Edit Schema feature, as described in Manage Schemas for Topics in the Control Center.
Note
A REST API call to compatibility mode is global on Confluent Platform, meaning it overrides any compatibility parameters set in Schema Registry properties files, as shown in the API usage example Update compatibility requirements globally.
To validate the compatibility of a given schema
You can validate the compatibility of a schema as follows:
On Confluent Cloud, Get compatibility level on a subject
On Confluent Platform, Get compatibility requirements on a subject
The following Confluent Schema Registry tutorials show examples of checking schema compatibility:
Format-specific compatibility rules for primitive types, objects, enums, arrays, and unions
Compatibility rules for each schema format are described in their related sections: