Schema Evolution and Compatibility for Schema Registry on Confluent Cloud

Ready to get started?

Schema evolution

Schema evolution is the practice of safely changing schemas over time while maintaining compatibility with existing producers and consumers. Schema Registry enforces compatibility by comparing new schema versions against previous versions using configurable compatibility types.

Compatibility types define which schema changes are allowed:

  • Backward - consumers using new schema can read data written with old schema (add optional fields, remove fields)

  • Forward - consumers using old schema can read data written with new schema (remove optional fields, add fields)

  • Full - both backward and forward compatible (add/remove optional fields only)

  • Transitive - compatibility checked against all previous versions, not just the last one

The default compatibility mode is BACKWARD. Each schema version gets a unique ID and an incremented version number. When a schema is updated, Schema Registry checks compatibility before accepting the new version.

To learn more about serdes for supported schema formats, see “Formats, Serializers, and Deserializers” in either the Confluent Cloud documentation or in the Confluent Platform documentation

Compatibility types

Summary

The following tables summarize the compatibility rules for the different schema formats: Avro, Protobuf, and JSON Schema for a given subject. The Confluent Schema Registry default compatibility type is BACKWARD. The sections that follow the tables describe all the compatibility types in more detail.

The allowed changes are dependent on how fields are originally defined. For example, as described under Backward compatibility, the ability to delete a field and keep the schema compatible requires that the field was either specified as optional or provided a default value in the original version. To learn more, see Configuration Options on connectors to self-managed Schema Registry that provide further control over compatibility requirements.

For JSON Schema, compatibility behavior depends on both the compatibility policy (compatibilityPolicy: LENIENT or STRICT) and the content model (additionalProperties: true for open, false for closed). The tables below show these in separate sections. A third policy, LOGICAL, applies across Avro, Protobuf, and JSON Schema.

Avro and Protobuf compatibility

Avro

Protobuf

Allowed changes

BW

FW

Full

BW

FW

Full

Add optional field

✔

✔

✔

✔

✔

✔

Remove optional field

✔

✔

✔

✔

✔

✔

Add required field

✔

Remove required field

✔

Add union/oneof variant

✔

✔

Remove union/oneof variant

✔

✔

Widen a scalar type

✔

✔

✔

✔

Narrow a scalar type

✔

✔

✔

✔

JSON Schema with lenient policy

Lenient policy allows schema changes regardless of the content model.

Allowed changes

BW

FW

Full

Add optional field

✔

✔

✔

Remove optional field

✔

✔

✔

Add required field

✔

Remove required field

✔

Add union/oneof variant

✔

Remove union/oneof variant

✔

Widen a scalar type

✔

Narrow a scalar type

✔

JSON Schema with strict policy

Strict policy compatibility rules vary by content model. Open content models (additionalProperties: true) allow more changes than closed models (additionalProperties: false).

Open content model (additionalProperties: true or omitted):

Allowed changes

BW

FW

Full

Add optional field

✔

Remove optional field

✔

Add required field

✔

Remove required field

✔

Add union/oneof variant

✔

Remove union/oneof variant

✔

Widen a scalar type

✔

Narrow a scalar type

✔

Closed content model (additionalProperties: false):

Allowed changes

BW

FW

Full

Add optional field

✔

Remove optional field

✔

Add required field

Remove required field

Add union/oneof variant

✔

Remove union/oneof variant

✔

Widen a scalar type

✔

Narrow a scalar type

✔

Table Legend:

  • ✔ = Compatible operation for the given schema format and compatibility type

  • Blank space = Incompatible operation for the given schema format and compatibility type

  • BW = Backward compatible

  • FW = Forward compatible

  • Full = Fully compatible (both backward and forward)

Note

Logical compatibility policy

The LOGICAL compatibility policy allows only the schema changes that Flink, Tableflow, and Lightning Tables all support. Use it on subjects that feed SQL queries or tables. The policy prevents a schema change from suspending a Tableflow table, failing a Flink statement, or causing existing records to be misread.

Tip

If a schema registration failed with a LOGICAL error, see Read and fix a LOGICAL error.

Schema Registry offers three compatibility policies: STRICT (the default), LENIENT, and LOGICAL.

  • STRICT and LENIENT check that Kafka consumers using Avro, Protobuf, or JSON Schema deserializers can read records written with a new schema version. The two differ only for JSON Schema. To learn more, see Compatibility types.

  • LOGICAL adds checks based on how Flink maps the schema to SQL types and how Tableflow and Lightning Tables map it to Apache Iceberg™ table types. Some changes pass the format’s own rules but still break these engines. For example, BACKWARD allows deleting a field. Tableflow doesn’t support dropping columns and suspends materialization for the topic. For the schema changes that Tableflow supports, see Schema compatibility and evolution.

The policy works alongside the subject’s compatibility type, such as BACKWARD. The compatibility type selects the earlier versions that Schema Registry compares a new schema with. The policy decides which rules the comparison uses.

LOGICAL applies to every subject it covers, even if Flink, Tableflow, or Lightning Tables doesn’t currently read that subject. If you need a change that LOGICAL blocks, you can override the policy for one registration with ?force=true. See Force a change that LOGICAL blocks.

How LOGICAL checks a schema

When you register a schema under LOGICAL, Schema Registry runs three kinds of checks. The registration fails if any of them fails.

  1. Format compatibility: the native Avro, Protobuf, or JSON Schema check for the subject’s compatibility type. For JSON Schema, this check uses LENIENT rules, because LOGICAL adds its own checks for the properties that become columns. Avro and Protobuf checks are the same under every policy.

  2. Change checks: compare the new schema with earlier versions and reject changes that break Flink, Tableflow, or Lightning Tables. See Changes that LOGICAL blocks.

  3. Validity checks: confirm that the schema itself maps to a SQL type and a table. These run on every registration, including the first version of a subject. See Schemas that LOGICAL rejects on every registration.

Before you begin

  • LOGICAL works only with the BACKWARD, BACKWARD_TRANSITIVE, FULL, and FULL_TRANSITIVE compatibility types. Tableflow and Lightning Tables support only backward-compatible evolution. Schema Registry rejects LOGICAL with NONE, FORWARD, or FORWARD_TRANSITIVE.

  • Turning on LOGICAL doesn’t re-check schemas that are already registered. Schema Registry checks new schemas against earlier versions, and the current schema must pass the validity checks. If the current schema contains something the validity checks reject, such as a self-referencing type, or the subject’s history contains a change that LOGICAL blocks, later registrations fail. Roll out LOGICAL shows how to check both.

  • While a subject uses LOGICAL, you can’t permanently delete the subject or its versions (error code 42205). To permanently delete them, first turn off LOGICAL for the subject.

  • For tables, the checks use the Iceberg type system. A Tableflow table backed by Delta Lake isn’t checked against Delta Lake’s type system.

  • Set the policy on the subject that holds the schema. If a topic’s subject is an alias, set it on the subject the alias points to. Registrations through the alias then follow that policy. A policy set on the alias itself has no effect, even though the alias’s configuration shows it.

Roll out LOGICAL

Turn on LOGICAL one subject at a time, starting with the subjects that feed Flink, Tableflow, or Lightning Tables. For subjects that feed Tableflow, use BACKWARD_TRANSITIVE. Each new schema is then checked against every earlier version that the table already holds.

The API examples use the SR_APIKEY, SR_APISECRET, and SCHEMA_REGISTRY_ENDPOINT shell variables, and jq. To find the Schema Registry values, see Setup and suggestions.

export SR_APIKEY=<sr_api_key>
export SR_APISECRET=<sr_api_secret>
export SCHEMA_REGISTRY_ENDPOINT=<sr_endpoint_url>
  1. Record the subject’s configuration to restore it later. Run the request twice: without defaultToGlobal to see what the subject sets itself, and with defaultToGlobal=true to see the settings in effect.

    curl --silent -u $SR_APIKEY:$SR_APISECRET \
      "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
    curl --silent -u $SR_APIKEY:$SR_APISECRET \
      "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>?defaultToGlobal=true"
    
  2. If the effective compatibilityLevel isn’t BACKWARD, BACKWARD_TRANSITIVE, FULL, or FULL_TRANSITIVE, or the subject feeds Tableflow and isn’t BACKWARD_TRANSITIVE, set it. For example:

    curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
      "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
    
  3. Test your current schema against the validity checks. Give an empty scratch subject the LOGICAL policy, and check the schema against it. The compatibility endpoint runs the validity checks even when a subject has no versions, and it doesn’t register anything:

    # Copy the latest schema, with its type and references
    curl --silent -u $SR_APIKEY:$SR_APISECRET \
      "$SCHEMA_REGISTRY_ENDPOINT/subjects/<subject_name>/versions/latest" \
      | jq '{schema, schemaType, references}' > scratch.json
    
    # Give the scratch subject the LOGICAL policy
    curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      --data '{"compatibility": "BACKWARD", "compatibilityPolicy": "LOGICAL"}' \
      "$SCHEMA_REGISTRY_ENDPOINT/config/logical-test"
    
    # Check the schema against it
    curl --silent -u $SR_APIKEY:$SR_APISECRET -X POST \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      --data @scratch.json \
      "$SCHEMA_REGISTRY_ENDPOINT/compatibility/subjects/logical-test/versions?verbose=true"
    
    # Remove the scratch configuration
    curl --silent -u $SR_APIKEY:$SR_APISECRET -X DELETE \
      "$SCHEMA_REGISTRY_ENDPOINT/config/logical-test"
    

    If is_compatible is false, fix the schema before you continue. See Schemas that LOGICAL rejects on every registration.

  4. Turn on LOGICAL, in the Cloud Console or with the API.

    In the Cloud Console, open the subject, click the edit icon next to Compatibility policy, select Logical, and click Save.

    With the API, send this request:

    curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      --data '{"compatibilityPolicy": "LOGICAL"}' \
      "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
    

    If the compatibility type isn’t supported, the request fails with error code 42203:

    compatibilityLevel='NONE' doesn't work with compatibilityPolicy='LOGICAL'.
    Use BACKWARD, BACKWARD_TRANSITIVE, FULL, or FULL_TRANSITIVE.
    
  5. Run the second request in step 1 again, and confirm that compatibilityPolicy is LOGICAL.

  6. Check the latest schema against the subject, with the request in Check schemas in CI. Under BACKWARD_TRANSITIVE and FULL_TRANSITIVE, this compares it with every earlier version. If a change check fails, the subject’s history already contains a change that LOGICAL blocks, and every later registration fails the same way. Use BACKWARD instead, which compares only with the latest version, or turn off LOGICAL for the subject.

  7. Add a LOGICAL check to your CI pipeline. See Check schemas in CI.

Set LOGICAL for the whole registry

To make LOGICAL the default for every subject that doesn’t set its own policy, send the same request to /config:

curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibilityPolicy": "LOGICAL"}' \
  "$SCHEMA_REGISTRY_ENDPOINT/config"

Warning

Every subject that inherits the setting is then checked, including subjects that no SQL engine reads. Registrations fail anywhere a schema can’t be represented as a table. Do this only after you roll out LOGICAL on individual subjects.

Check schemas in CI

The compatibility endpoint runs the same checks as registration, without registering anything. Without a version number, it compares the schema with the versions that the subject’s compatibility type selects, the same ones registration uses.

Put the request body in a file, for example schema.json. Let jq escape the schema for you:

# Avro
jq -n --rawfile s order.avsc '{schema: $s}' > schema.json
# Protobuf
jq -n --rawfile s order.proto '{schemaType: "PROTOBUF", schema: $s}' > schema.json
# JSON Schema
jq -n --rawfile s order.json '{schemaType: "JSON", schema: $s}' > schema.json

If the schema uses references, add them to the body in references, as you do when you register it.

The endpoint returns HTTP 200 for compatible and incompatible schemas. Fail the build on the is_compatible field. Other status codes, such as 422 for an unparsable schema, mean the check didn’t run:

status=$(curl --silent -o result.json -w '%{http_code}' \
  -u $SR_APIKEY:$SR_APISECRET -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @schema.json \
  "$SCHEMA_REGISTRY_ENDPOINT/compatibility/subjects/<subject_name>/versions?verbose=true")
if [ "$status" != "200" ]; then cat result.json; exit 1; fi
jq -e '.is_compatible' result.json > /dev/null \
  || { jq -r '.messages[]' result.json; exit 1; }

When the check fails, messages lists each problem in the same form as a registration error. See Read and fix a LOGICAL error.

A change you plan to force fails this check too. Run the ?force=true registration as a separate step that someone reviews and approves.

Changes that LOGICAL allows

Under BACKWARD and BACKWARD_TRANSITIVE, you can evolve a schema in these ways:

  • Add an optional field.

  • Make a NOT NULL field nullable. In Protobuf, mark it optional. In JSON Schema, remove it from required.

  • Widen an integer, for example INT to BIGINT (Avro int to long, Protobuf int32 to int64). JSON Schema integer has no width. This change doesn’t apply to it.

  • Increase a decimal’s precision while keeping its scale.

  • Increase the length of a VARCHAR or VARBINARY.

  • Add an enum symbol.

Under FULL and FULL_TRANSITIVE, LOGICAL also checks each change in reverse. It rejects widening a type, increasing a decimal’s precision or a length, making a field nullable, and adding an enum symbol. Adding an optional field is the only change in the preceding list that FULL allows.

Changes that LOGICAL blocks

The tables in this section list each change that LOGICAL rejects, where it breaks, how to make the change safely, and the rule name that appears as errorType in the error.

  • Breaks lists the engines that the change affects when it reaches them: Tableflow suspends materialization for the topic, a Flink statement fails, or, for type changes, the engine reads existing values incorrectly with no error. Any query or job that references a changed field still needs updating.

  • The examples use Logical Type (SQL) names. To see how each type maps to Avro, Protobuf, and JSON Schema, see Logical Type mappings. For what nullable and optional mean in each format, see Nullable and optional fields in each format.

Fields

Change

What goes wrong

Breaks

How to fix

Rule

Delete a field, including a field with a default value

Tableflow doesn’t support dropping columns. It suspends materialization for the topic.

Tableflow, Lightning Tables

Keep the field, and retire it instead. See Retire a field.

FIELD_DELETED

Rename a field

Renaming a field isn’t supported, including with Avro aliases. Schema Registry treats a rename as deleting the field and adding a new one.

Tableflow, Lightning Tables

Add the new name as an optional field, then retire the old one. See Rename a field.

FIELD_DELETED

Add a required field, meaning any field that isn’t nullable, even if it has a default value

Existing records have no value for the field.

Tableflow, Lightning Tables

Add the field as optional. See Nullable and optional fields in each format.

REQUIRED_FIELD_ADDED

Note

LOGICAL doesn’t check field order in Confluent Cloud. A Tableflow table can’t apply a new column order. Keep the original order and add new fields at the end.

Types

Change

What goes wrong

Breaks

How to fix

Rule

Change a field to an unrelated type

A string field becomes an integer.

Flink, Tableflow, Lightning Tables

Keep the type. Add a new field with the new type, and retire the old one.

UNSUPPORTED_TYPE_CHANGE

Narrow a type so it can’t hold all existing values

Protobuf int64 to int32. Readers silently truncate values that don’t fit in 32 bits.

Flink, Tableflow, Lightning Tables

Keep the wider type, and check the range in the producer.

UNSUPPORTED_TYPE_CHANGE

Promote an integer to a floating-point type

Avro int or long to float or double, or JSON Schema integer to number. Large values can round to the nearest representable value, and tables can’t change an integer column to a floating-point one.

Flink, Tableflow, Lightning Tables

Keep the integer type, or add a new floating-point field.

UNSUPPORTED_TYPE_CHANGE

Turn a date into a timestamp

Avro date to timestamp-millis. The day count 19,723 reads as about 19.7 seconds after the Unix epoch.

Flink, Tableflow, Lightning Tables

Keep the date field, and add a new timestamp field.

UNSUPPORTED_TYPE_CHANGE

Change time or timestamp precision

Avro timestamp-millis to timestamp-micros, in either direction. Existing values are off by a factor of 1,000.

Flink

Keep the original precision.

UNSUPPORTED_TYPE_CHANGE

Change a decimal’s scale, or reduce its precision

DECIMAL(10, 2) to DECIMAL(10, 3). The stored value 1234 reads as 1.234, but it should read as 12.34.

Flink, Tableflow, Lightning Tables

Keep the scale. Under BACKWARD or BACKWARD_TRANSITIVE, you can increase the precision.

UNSUPPORTED_TYPE_CHANGE

Shorten a VARCHAR or VARBINARY length

VARCHAR(100) to VARCHAR(50). Longer existing values no longer fit.

Flink

Keep the original length. Under BACKWARD or BACKWARD_TRANSITIVE, you can increase the length.

UNSUPPORTED_TYPE_CHANGE

Change a fixed length

BINARY(16) to BINARY(32), or a change to a CHAR length.

Flink, Tableflow, Lightning Tables

Keep the original length.

UNSUPPORTED_TYPE_CHANGE

Change a map’s key type

MAP<STRING, INT> to MAP<INT, INT>.

Flink, Tableflow, Lightning Tables

Keep the key type. Add a new map field if you need a different key.

MAP_KEY_TYPE_MISMATCH, and UNSUPPORTED_TYPE_CHANGE for Flink

Nullability, defaults, and enums

Change

What goes wrong

Breaks

How to fix

Rule

Make a nullable field NOT NULL, or make the items of an array or map NOT NULL

Existing records can contain null values.

Flink, Tableflow, Lightning Tables

Keep the field nullable, and require a value in the producer instead.

NULLABLE_TO_NON_NULLABLE

Remove the default value from a NOT NULL field

Flink uses the default when it reads records that don’t contain the field.

Flink

Restore the default value.

NON_NULLABLE_DEFAULT_REMOVED

Remove an enum symbol

Existing records can use the symbol.

Tableflow, Lightning Tables

Keep every existing symbol, and stop producing the one you want to retire.

ENUM_DELETED

Nullable and optional fields in each format

Format

A field is nullable when

Add an optional field

Avro

Its type is a union with null.

{"name": "notes", "type": ["null", "string"], "default": null}

Protobuf

It’s marked optional in proto3, or uses a wrapper type such as google.protobuf.Int64Value. Any other proto3 field without optional is NOT NULL.

Add a field with a new tag number, marked optional. A new field without optional fails with REQUIRED_FIELD_ADDED.

JSON Schema

It isn’t listed in required. Adding an existing property to required fails with NULLABLE_TO_NON_NULLABLE, and adding a new property that’s listed in required fails with REQUIRED_FIELD_ADDED.

Add the property without listing it in required.

Logical Type (SQL), see Logical Type mappings

It’s declared without NOT NULL.

Declare the field without NOT NULL.

Retire a field

Because LOGICAL blocks deleting a field, retire it instead:

  1. If the field is NOT NULL, make it nullable. BACKWARD and BACKWARD_TRANSITIVE allow this change. FULL and FULL_TRANSITIVE don’t allow it. Under those types, change the subject to BACKWARD_TRANSITIVE first.

  2. Update producers to stop writing the field. New records then have no value for it.

  3. Update Flink statements and other consumers to stop reading the field.

The column stays in the table and holds null for new records.

Rename a field
  1. Add the new name as an optional field.

  2. Update producers to write both fields.

  3. Move Flink statements and other consumers to the new field.

  4. Retire the old field.

Records written before the change have null in the new column, and their values stay in the old column. Until you backfill, read both, for example COALESCE(new_name, old_name).

Schemas that LOGICAL rejects on every registration

These checks look at the schema by itself. If a registered schema fails one of them, fixing it can require a change that LOGICAL blocks. Register the fix with ?force=true. Under transitive compatibility types, later registrations can need it too. See Force a change that LOGICAL blocks.

Schema contains

Limit

Breaks

Rule

A type that refers to itself

Not supported

Flink, Tableflow, Lightning Tables

CYCLIC_TYPE

A reference to a type that the schema doesn’t define

Not supported

Flink, Tableflow, Lightning Tables

UNRESOLVED_TYPE_REF

A decimal precision or scale outside the supported range

Precision 1 to 38, scale 0 to the precision

Flink, Tableflow, Lightning Tables

PRECISION_OUT_OF_RANGE, SCALE_OUT_OF_RANGE

A CHAR, BINARY, or other fixed length below the lower limit

At least 1

Flink, Tableflow, Lightning Tables

LENGTH_OUT_OF_RANGE

Nesting deeper than the supported number of levels

100 levels

Flink, Tableflow, Lightning Tables

UNREPRESENTABLE_TYPE

A record, message, or union with no fields or types

At least one

Tableflow, Lightning Tables

EMPTY_STRUCT

Two field names that differ only by letter case, such as id and ID

Not supported

Tableflow, Lightning Tables

FIELD_NAME_CASE_COLLISION

A VARIANT type

Not supported

Tableflow, Lightning Tables

UNREPRESENTABLE_TYPE

A timestamp with precision greater than 6

6 (microseconds)

Tableflow, Lightning Tables

UNREPRESENTABLE_TYPE

Read and fix a LOGICAL error

When a registration fails, Schema Registry returns HTTP 409 with error_code 40901 and a message. The message lists each problem with its rule in errorType and a description that names the field. For example, deleting the carrier field from a BACKWARD subject returns:

{
  "error_code": 40901,
  "message": "Schema being registered is incompatible with an earlier schema for subject \"shipment-orders\", details: [{errorType:\"FIELD_DELETED\", category:[\"ICEBERG_V2\"], description:\"Schema no longer includes field 'carrier'. Restore the field.\", additionalInfo:\"carrier\"}, {validateFields: \"true\", compatibility: \"BACKWARD\"}]"
}

The Cloud Console shows the same message below the schema editor. The entries inside details aren’t strict JSON. To detect a rule in a script, match on the text errorType:"<RULE>". Entries from LOGICAL include a category that names the engines the change breaks (FLINK, ICEBERG_V2, or both). Entries without a category come from the format’s own compatibility check.

Find the errorType in the following table. Rules not listed here come from the format’s own compatibility check, which ?force=true can’t bypass.

errorType

Breaks

How to fix

Details

CYCLIC_TYPE, UNRESOLVED_TYPE_REF, PRECISION_OUT_OF_RANGE, SCALE_OUT_OF_RANGE, LENGTH_OUT_OF_RANGE, UNREPRESENTABLE_TYPE

Flink, Tableflow, Lightning Tables (UNREPRESENTABLE_TYPE for VARIANT or timestamps: Tableflow, Lightning Tables only)

Change the schema so it stays within the supported limits. If a registered version already fails the check, register the fix with ?force=true.

Schemas that LOGICAL rejects on every registration

EMPTY_STRUCT, FIELD_NAME_CASE_COLLISION

Tableflow, Lightning Tables

Add a field, or rename one of the colliding fields. If a registered version already fails the check, register the fix with ?force=true.

Schemas that LOGICAL rejects on every registration

ENUM_DELETED

Tableflow, Lightning Tables

Keep every existing symbol.

Changes that LOGICAL blocks

FIELD_DELETED

Tableflow, Lightning Tables

Add the field back, then retire or rename it.

Retire a field, Rename a field

MAP_KEY_TYPE_MISMATCH

Tableflow, Lightning Tables (Flink reports UNSUPPORTED_TYPE_CHANGE for the same change)

Keep the map’s key type.

Changes that LOGICAL blocks

NON_NULLABLE_DEFAULT_REMOVED

Flink

Restore the default value.

Changes that LOGICAL blocks

NULLABLE_TO_NON_NULLABLE

Flink, Tableflow, Lightning Tables

Keep the field nullable.

Changes that LOGICAL blocks

REQUIRED_FIELD_ADDED

Tableflow, Lightning Tables

Add the field as optional.

Nullable and optional fields in each format

UNSUPPORTED_TYPE_CHANGE, TYPE_MISMATCH

Depends on the change. See the Breaks column in Changes that LOGICAL blocks.

Keep the original type, or add a new field with the new type.

Changes that LOGICAL blocks

Force a change that LOGICAL blocks

If you need a change that LOGICAL blocks, and you accept that it can break Flink, Tableflow, or Lightning Tables, add ?force=true to the registration request:

curl --silent -u $SR_APIKEY:$SR_APISECRET -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @schema.json \
  "$SCHEMA_REGISTRY_ENDPOINT/subjects/<subject_name>/versions?force=true"

Anyone who can register schemas in the subject can use force. They don’t need another role, and you can’t turn force off for a subject or the registry.

force skips all LOGICAL checks, both the change checks and the validity checks, for that one registration. It doesn’t skip the format compatibility check. force can’t register a schema that the format’s own rules reject.

To decide whether a change is safe to force, check its Breaks column in Changes that LOGICAL blocks or Schemas that LOGICAL rejects on every registration. For example, if only Flink reads a subject, a change that breaks only Tableflow and Lightning Tables doesn’t affect it today, but it does affect any table you create from the subject later.

Kafka consumers don’t get deserialization errors after a forced change, but they can read wrong values with no error. For example:

  • After a forced int64 to int32 change, consumers using the new schema silently truncate large values.

  • After a forced change that removes optional from a proto3 field, consumers using the new schema read an unset value as the type’s default (0, "", or false). They don’t read it as null.

Before you force a change:

  • Plan for the engines it breaks. For example, after you force a field deletion, Tableflow suspends materialization for the topic, because it doesn’t support dropping columns. Update the Flink statements that read the subject. See Schema compatibility and evolution.

  • Under BACKWARD_TRANSITIVE and FULL_TRANSITIVE, every later version is also compared with the versions before the forced one. A forced change such as a field deletion then fails every later registration. Each one also needs ?force=true, which skips all LOGICAL checks for it. Your CI check also fails for each one.

Turn off LOGICAL

Restore what you recorded in step 1 of Roll out LOGICAL:

  • If the subject set its own policy, set compatibilityPolicy back to that value. STRICT is the default.

    curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      --data '{"compatibilityPolicy": "STRICT"}' \
      "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
    
  • If the subject inherited its policy from the registry, remove the subject’s own policy. The subject keeps its compatibility type and inherits the registry’s policy again:

    curl --silent -u $SR_APIKEY:$SR_APISECRET -X PUT \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      --data '{"compatibilityPolicy": null}' \
      "$SCHEMA_REGISTRY_ENDPOINT/config/<subject_name>"
    

    Don’t use DELETE /config/<subject_name> for this. It also removes the subject’s compatibility type.

  • If you changed the compatibility type in step 2, set compatibility back to the value you recorded.

  • To turn off a registry-wide LOGICAL setting, send the first request in this list to /config.

In the Cloud Console, select Strict or Lenient in the Compatibility policy dialog. If you added a LOGICAL check to your CI pipeline, remove it.

Avro, Protobuf, and JSON Schema have different compatibility rules

Schema evolution and compatibility rules vary somewhat based on schema format. The scenarios and source code examples given on this page are geared for Avro, which was the first serializer and deserializer that Confluent supported. Avro was developed with schema evolution in mind, and its specification clearly states the rules for backward compatibility. The rules and grammar for JSON Schema and Protobuf can be more nuanced.

Although the general concepts apply to all formats, the details on how compatibility is implemented differs depending on whether you use Avro, JSON Schema, or Protobuf. For more in-depth explanations and specific examples on the newer formats, see the following blog posts and documentation:

Protobuf

Note that best practice for Protobuf is to use BACKWARD_TRANSITIVE, as adding new message types is not forward compatible.

JSON Schema

Backward compatibility

BACKWARD compatibility means that consumers using the new schema can read data produced with the last schema. For example, if there are three schemas for a subject that change in order X-2, X-1, and X then BACKWARD compatibility ensures that consumers using the new schema X can process data written by producers using schema X or X-1, but not necessarily X-2. If the consumer using the new schema needs to be able to process data written by all registered schemas, not just the last two schemas, then use BACKWARD_TRANSITIVE instead of BACKWARD. For example, if there are three schemas for a subject that change in order X-2, X-1, and X then BACKWARD_TRANSITIVE compatibility ensures that consumers using the new schema X can process data written by producers using schema X, X-1, or X-2.

  • BACKWARD: consumer using schema X can process data produced with schema X or X-1

  • BACKWARD_TRANSITIVE: consumer using schema X can process data produced with schema X, X-1, or X-2

Important

  • The Confluent Schema Registry default compatibility type is BACKWARD, not BACKWARD_TRANSITIVE. The main reason that BACKWARD compatibility mode is the default, and preferred for Kafka, is so that you can rewind consumers to the beginning of the topic. With FORWARD compatibility mode, you aren’t guaranteed the ability to read old messages. Also, FORWARD compatibility mode is harder to work with. In a sense, you need to anticipate all future changes. For example, in FORWARD compatibility mode with Protobuf, you can’t add new message types to a schema.

  • For Kafka Streams, only BACKWARD compatibility is supported. To learn more, see the note about Kafka Streams under Order of Upgrading Clients.

  • To understand how Avro uses a newer backward-compatible schema to decode (deserialize) data that was encoded (serialized) with an older schema, see ResolvingDecoder in the Apache Avro project.

An example of a backward compatible change is removing a field. A consumer developed to process events without this field can process events written with the old schema, which contain the field. The consumer ignores that field.

Consider the case where all of the data in Kafka is also loaded into HDFS, and you want to run SQL queries (for example, using Apache Hive) over all the data. Here, it is important that the same SQL queries continue to work even as the data is undergoing changes over time. To support this kind of use case, you can evolve the schemas in a backward compatible way. All Formats, Serializers, and Deserializers have rules as to what changes are allowed in the new schema for it to be backward compatible. For example, here are the Avro rules for compatibility. If all schemas are evolved in a backward compatible way, you can always use the latest schema to query all the data uniformly.

For example, an application can evolve the user schema from the previous section to the following by adding a new field favorite_color:

{"namespace": "example.avro",
 "type": "record",
 "name": "user",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": "int"},
     {"name": "favorite_color", "type": "string", "default": "green"}
 ]
}

Note that the new field favorite_color has the default value “green”. This allows data encoded with the old schema to be read with the new one. The default value specified in the new schema is used for the missing field when deserializing the data encoded with the old schema. Had the default value been omitted in the new field, the new schema would not be backward compatible with the old one since it’s not clear what value should be assigned to the new field, which is missing in the old data.

Forward compatibility

FORWARD compatibility means that data produced with a new schema can be read by consumers using the last schema, even though they may not be able to use the full capabilities of the new schema. For example, if there are three schemas for a subject that change in order X-2, X-1, and X then FORWARD compatibility ensures that data written by producers using the new schema X can be processed by consumers using schema X or X-1, but not necessarily X-2. If data produced with a new schema needs to be read by consumers using all registered schemas, not just the last two schemas, then use FORWARD_TRANSITIVE instead of FORWARD. For example, if there are three schemas for a subject that change in order X-2, X-1, and X then FORWARD_TRANSITIVE compatibility ensures that data written by producers using the new schema X can be processed by consumers using schema X, X-1, or X-2.

  • FORWARD: data produced using schema X can be read by consumers with schema X or X-1

  • FORWARD_TRANSITIVE: data produced using schema X can be read by consumers with schema X, X-1, or X-2

An example of a forward compatible schema modification is adding a new field. In most data formats, consumers that were written to process events without the new field can continue doing so even when they receive new events that contain the new field.

Consider a use case where a consumer has application logic tied to a particular version of the schema. When the schema evolves, the application logic may not be updated immediately. Therefore, you need to be able to project data with newer schemas onto the (older) schema that the application understands. To support this use case, you can evolve the schemas in a forward compatible way: data encoded with the new schema can be read with the old schema. For example, the new user schema shown in the previous section on backward compatibility is also forward compatible with the old one. When projecting data written with the new schema to the old one, the new field is simply dropped. Had the new schema dropped the original field favorite_number (number, not color), it would not be forward compatible with the original user schema since consumers wouldn’t know how to fill in the value for favorite_number for the new data because the original schema did not specify a default value for that field.

Full compatibility

FULL compatibility means schemas are both backward and forward compatible. Schemas evolve in a fully compatible way: old data can be read with the new schema, and new data can also be read with the last schema. For example, if there are three schemas for a subject that change in order X-2, X-1, and X then FULL compatibility ensures that consumers using the new schema X can process data written by producers using schema X or X-1, but not necessarily X-2, and that data written by producers using the new schema X can be processed by consumers using schema X or X-1, but not necessarily X-2. If the new schema needs to be forward and backward compatible with all registered schemas, not just the last two schemas, then use FULL_TRANSITIVE instead of FULL. For example, if there are three schemas for a subject that change in order X-2, X-1, and X then FULL_TRANSITIVE compatibility ensures that consumers using the new schema X can process data written by producers using schema X, X-1, or X-2, and that data written by producers using the new schema X can be processed by consumers using schema X, X-1, or X-2.

  • FULL: backward and forward compatible between schemas X and X-1

  • FULL_TRANSITIVE: backward and forward compatible between schemas X, X-1, and X-2

In Avro and Protobuf, you can define fields with default values. In that case, adding or removing a field with a default value is a fully compatible change.

In Avro, a field is made nullable (and therefore “optional” at the schema level) by using a union type that includes "null" (for example, ["null", "string"]). Providing a default value is not required for nullability itself, but it is important for schema evolution when adding a new field so that existing data can still be read. Avro requires that the default value conform to the first branch of the union, so it is common to put "null" first and use "default": null for fields that are intended to be optional for schema evolution. Note that Avro does not have a built-in optional keyword like Protobuf; instead, the union-with-null pattern serves this purpose.

Compatibility rules for supported schema types are described in Compatibility checks.

JSON Schema does not explicitly define compatibility rules, so this blog post further explains how JSON Schema compatibility works, including full compatibility.

No compatibility checking

NONE compatibility type means schema compatibility checks are disabled.

Sometimes you need to make incompatible changes. For example, modifying a field type from Number to String. In this case, you need to either upgrade all producers and consumers to the new schema version at the same time or, more likely, create a brand-new topic and start migrating applications to use the new topic and new schema, avoiding the need to handle two incompatible versions in the same topic.

Transitive property

Transitive compatibility checking is important once you have more than two versions of a schema for a given subject. If compatibility is configured as transitive, then it checks compatibility of a new schema against all previously registered schemas; otherwise, it checks compatibility of a new schema only against the latest schema.

For example, if there are three schemas for a subject that change in order X-2, X-1, and X then:

  • transitive: ensures compatibility between X-2 <==> X-1 and X-1 <==> X and X-2 <==> X

  • non-transitive: ensures compatibility between X-2 <==> X-1 and X-1 <==> X, but not necessarily X-2 <==> X

Refer to an example of schema changes which are incrementally compatible, but not transitively so.

The Confluent Schema Registry default compatibility type BACKWARD is non-transitive, which means that it’s not BACKWARD_TRANSITIVE. As a result, new schemas are checked for compatibility only against the latest schema.

Order of upgrading clients

The configured compatibility type has an implication on the order for upgrading client applications, that is, the producers using schemas to write events to Kafka and the consumers using schemas to read events from Kafka. Depending on the compatibility type:

  • BACKWARD or BACKWARD_TRANSITIVE: there is no assurance that consumers using older schemas can read data produced using the new schema. Therefore, upgrade all consumers before you start producing new events.

  • FORWARD or FORWARD_TRANSITIVE: there is no assurance that consumers using the new schema can read data produced using older schemas. Therefore, first upgrade all producers to using the new schema and make sure the data already produced using the older schemas are not available to consumers, then upgrade the consumers.

  • FULL or FULL_TRANSITIVE: there are assurances that consumers using older schemas can read data produced using the new schema and that consumers using the new schema can read data produced using older schemas. Therefore, you can upgrade the producers and consumers independently.

  • NONE: compatibility checks are disabled. Therefore, you need to be cautious about when to upgrade clients.

Important

  • For Kafka Streams only FULL, TRANSITIVE, and BACKWARD compatibility is supported. For a plain consumer, it is safe to upgrade the consumer to the new schema after the producer is upgraded because a plain consumer reads only from the input topic. For Kafka Streams, the scenario is different. When you upgrade Kafka Streams, it also can read from the input topic (that now contains data with the new schema). However, in contrast to a plain consumer, Kafka Streams must also be able to read the old schema (from the state or changelog). Therefore, only BACKWARD compatibility is supported. The Kafka Streams apps must be upgraded first, then it’s safe to upgrade the upstream producer that writes into the input topic. FULL, FULL_TRANSITIVE, and BACKWARD_TRANSITIVE compatibilities are always supported for Kafka Streams, as they include backward compatibility and so are, in effect, “stronger” settings than BACKWARD.

  • With Confluent Cloud for Apache Flink®, FULL, FULL_TRANSITIVE, and BACKWARD_TRANSITIVE compatibility types are supported. For more information, see Schema and Statement Evolution with Flink.

Specify schema compatibility requirements per subject

You can configure and update schema compatibility requirements globally or on a per-subject basis. To learn more, see Compatibility in the Schema Registry API Reference and the following topics in the Schema Registry API Usage Examples and Reference docs:

On Confluent Cloud:

On Confluent Platform:

Examples

Each of the sections above has an example of the compatibility type. An additional reference for Avro is Avro compatibility test suite, which presents multiple test cases with two schemas and the respective result of the compatibility test between them.

Using compatibility types

Compatibility rules and references for all supported schema types are described in Compatibility checks in Formats, Serializers, and Deserializers, and in Schema Evolution and Compatibility (on Confluent Cloud) and Schema Evolution and Compatibility (on Confluent Platform).

For details on how to use Schema Registry to store schemas and enforce compatibility rules during schema evolution, see the REST API usage examples and references for Confluent Cloud and Confluent Platform:

Here are some tips to get you started.

To check the currently configured global compatibility type

On Confluent Cloud:

On Confluent Platform:

To set the global or per schema compatibility level

You can configure schema compatibility in the following ways:

On Confluent Cloud:

On Confluent Platform:

Note

A REST API call to compatibility mode is global on Confluent Platform, meaning it overrides any compatibility parameters set in Schema Registry properties files, as shown in the API usage example Update compatibility requirements globally.

To validate the compatibility of a given schema

You can validate the compatibility of a schema as follows:

The following Confluent Schema Registry tutorials show examples of checking schema compatibility:

Format-specific compatibility rules for primitive types, objects, enums, arrays, and unions

Compatibility rules for each schema format are described in their related sections: