# Logical Types Schema Registry supports Logical Types, which let you represent a schema using SQL types, not the syntax of a serialization format. For example, this Avro schema: ```json { "type": "record", "name": "Person", "fields": [ {"name": "id", "type": "string"}, {"name": "name", "type": "string"}, {"name": "age", "type": ["null", "int"]}, {"name": "active", "type": "boolean"} ] } ``` becomes this Logical Type: ```text STRUCT Person ( id STRING NOT NULL, name STRING NOT NULL, age INT, active BOOLEAN NOT NULL ) ``` Register the Logical Type the same way as any other schema: ```bash curl -X POST -H "Content-Type: application/json" \ --data '{ "schemaType": "AVRO", "schema": "STRUCT Person ( id STRING NOT NULL, name STRING NOT NULL, age INT, active BOOLEAN NOT NULL )" }' \ 'http://localhost:8081/subjects/mytopic-value/versions' ``` `schemaType` still must be `AVRO`, `PROTOBUF`, or `JSON`, because this indicates which serialization format clients use. If you retrieve the preceding schema, it matches the Avro form shown earlier. Specifying `PROTOBUF` instead returns: ```text syntax = "proto3"; message Person { string id = 1; string name = 2; optional int32 age = 3; bool active = 4; } ``` And specifying `JSON` returns: ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "additionalProperties": false, "properties": { "id": {"type": "string"}, "name": {"type": "string"}, "age": {"oneOf": [{"type": "null"}, {"type": "number"}]}, "active": {"type": "boolean"} }, "required": ["id", "name", "active"], "title": "Person", "type": "object" } ``` ## Retrieve a schema in Logical Type form To retrieve the Logical Type form of a schema, pass `?format=LOGICAL` on the request: ```bash curl -H "Content-Type: application/json" \ 'http://localhost:8081/subjects/mytopic-value/versions/1?format=LOGICAL' ``` Every endpoint that returns a schema accepts the `format` parameter, not only version reads: - Registering a schema - Subject and version lookups - The by-ID endpoint under `/schemas` - The by-GUID endpoint under `/schemas` Input works the other way: Schema Registry detects a Logical Type from the request body. It first tries to parse the content as the declared `schemaType` and falls back to Logical Type syntax only if that parse fails. Because detection is content-driven, you can send a Logical Type anywhere you can send a schema, including the compatibility endpoints for a single version (`/compatibility/subjects//versions/`) and for all versions (`/compatibility/subjects//versions`). This flexibility lets you check whether Schema Registry would accept a Logical Type before you register it: ```bash curl -X POST -H "Content-Type: application/json" \ --data '{ "schemaType": "AVRO", "schema": "STRUCT Person ( id STRING NOT NULL, name STRING NOT NULL, age BIGINT, active BOOLEAN NOT NULL )" }' \ 'http://localhost:8081/compatibility/subjects/mytopic-value/versions/latest' ``` ## Namespaces, ENUM, and nested STRUCT Logical Types can represent most of what any of the serialization formats can express. This example declares a `NAMESPACE`, an `ENUM`, and two `STRUCT` types, one of which references the other: ```text NAMESPACE com.example.demo; ENUM Status ('ACTIVE', 'PAUSED', 'CANCELLED'); STRUCT Address ( street STRING NOT NULL, city STRING NOT NULL, postal_code VARCHAR(16) ); STRUCT Order ( id STRING NOT NULL, retries INT NOT NULL DEFAULT 0, status Status NOT NULL, shipping Address, billing Address ); ``` An `ENUM` symbol is a quoted string literal, so the grammar itself allows any character. Schema Registry doesn’t check symbol characters against the target format at parse time. It passes each symbol straight to that format’s own schema builder, which enforces its own naming rules. Avro and Protobuf both require symbols to be valid identifiers, so a symbol like `'in-progress'` fails at registration for either one. JSON Schema has no such restriction. For Protobuf, symbols number in declaration order starting at 0. To set a symbol’s number explicitly, add a `WITH` clause after that symbol, for example `ENUM Status ('UNKNOWN' WITH ('protobuf.enum.number' = '0'), 'ACTIVE' WITH ('protobuf.enum.number' = '1'))`. proto3 requires the first declared symbol to be number 0, and every number must be unique. Number 0 is also proto3’s default value for an unset enum field. An unset field then silently reads as the symbol numbered 0, `ACTIVE` in the `Order` example’s `Status`, even though `status` is `NOT NULL`. Declare a sentinel symbol first, such as `'UNKNOWN'`, if that default matters to readers of the data. Because numbering follows declaration order, inserting, removing, or reordering symbols renumbers every symbol after the change. Existing Protobuf-encoded data then decodes as different symbols. Add new symbols only at the end and keep existing ones, or number every symbol explicitly. This also applies to a sentinel symbol: declare it in the first version, because adding it first to an existing `ENUM` renumbers every other symbol. When a script declares more than one named type, Schema Registry infers the root schema as a declared type that no other declared type references. In this example, that’s `Order`, because `Status` and `Address` are both referenced by it. When more than one declared type has no incoming references, the first one declared becomes the root. The rest remain as peer named types in Protobuf and JSON Schema, which both support more than one named type per schema. Avro allows only one top-level type, so the Avro output entirely drops a declared type that the root doesn’t reference. When every declared type has an incoming reference from another declared type, forming a cycle, Schema Registry can’t infer a root and rejects the script. To set the root explicitly instead of relying on this inference, add a trailing `TYPE Order NOT NULL;` statement after the declarations. If the inferred root is an `ENUM`, registering it for Protobuf fails, because Protobuf has no message-less root. Use the `ENUM` as a field of a `STRUCT` and let that `STRUCT` be the root, by inference or with `TYPE`. This `Order` example corresponds to the following Avro schema. The Avro form is more verbose, and nullable fields translate to unions, as Avro requires: ```json { "type": "record", "name": "Order", "namespace": "com.example.demo", "fields": [ {"name": "id", "type": "string"}, {"name": "retries", "type": "int", "default": 0}, {"name": "status", "type": { "type": "enum", "name": "Status", "symbols": ["ACTIVE", "PAUSED", "CANCELLED"] }}, {"name": "shipping", "type": ["null", { "type": "record", "name": "Address", "fields": [ {"name": "street", "type": "string"}, {"name": "city", "type": "string"}, {"name": "postal_code", "type": ["null", {"type": "string", "flink.maxLength": 16}]} ] }]}, {"name": "billing", "type": ["null", "Address"]} ] } ``` Logical Types can be user-defined (`Status`, `Order`, and `Address` in the preceding example, using `STRUCT` or `ENUM`) or built-in, corresponding to SQL types such as `INT`. To register a built-in type on its own, such as when using a simple integer as a message key’s schema, use `TYPE`: ```text TYPE INT NOT NULL ``` This declaration corresponds to `{"type": "int"}` in Avro. Types are nullable by default, so specify `NOT NULL` to get a non-nullable type. Without it, `TYPE INT` corresponds to `["null", "int"]`. Logical Types can also express unions: ```text STRUCT IdAndPayload ( id STRING NOT NULL, payload UNION ( text STRING NOT NULL, number BIGINT NOT NULL, blob BINARY(16) NOT NULL ) NOT NULL ); ``` A `UNION` translates to an Avro union, a Protobuf `oneof`, or a JSON Schema `oneOf`. Every branch in the preceding example is `NOT NULL` because a nullable branch would become a nested union, and Avro doesn’t support that. Avro also rejects a union with two unnamed branches of the same Avro type. Each branch must map to a distinct Avro type, not only use a distinct name or Logical Type. For example, `INT`, `SMALLINT`, and `DATE` all map to Avro `int`, and `CHAR(n)`, `VARCHAR(n)`, and `STRING` all map to `string`. The same rule covers collections, whatever their element types. An Avro union can hold at most one `array` and one `map`. `ARRAY`, `MULTISET`, and a `MAP` with a non-string key all become Avro arrays, so any two of them conflict. For Avro, Schema Registry rejects such a schema at registration. It doesn’t silently collapse or rename a branch. JSON Schema has no matching check. Two branches that map to the same JSON type, such as `text STRING` and `blob BINARY(16)` in the preceding example (both become `string`), register without error. `oneOf` passes only when exactly one branch matches a value, so a JSON Schema validator rejects every string value for that `UNION`, because a string always matches both branches. `number` values, such as the `number BIGINT` branch, still validate. The `confluent:union` branch names in the schema don’t prevent this. When you register a `UNION` as `JSON`, make sure each branch maps to a distinct JSON type. The `UNION` itself can also be nullable, unlike its branches. For example, `payload UNION (...)` without `NOT NULL` doesn’t nest a `["null", [...]]` union, which Avro forbids. Schema Registry splices `null` into the existing branch list instead, producing `["null", "string", "long", {"type": "fixed", "name": "blob", "size": 16}]` for the preceding example. A `BINARY(n)` branch, such as `blob` in the example, becomes an Avro `fixed` type named after the branch, here `blob`. This name isn’t qualified by the enclosing struct or union. If two different unions in the same schema each declare a same-named `BINARY(n)` branch with different sizes, Schema Registry rejects the schema at registration. The two conflict under the same Avro fixed-type name. ## Schema references for Logical Types Logical Types can refer to other named types. Schema Registry stores every Logical Type in its converted native form, the same way it stores any other schema, so the referenced schema must use the same `schemaType` as the referencing one. It can be registered either as a Logical Type or as a native schema of that type. If a type name isn’t built-in or locally defined as a `STRUCT` or `ENUM`, Schema Registry treats it as a reference to an external Logical Type. This also applies to a misspelled built-in or local type name, for example `BIGIN` for `BIGINT`. Schema Registry doesn’t reject it as an unknown type. It treats the misspelling as an external reference, so it fails with a missing- or unresolved-reference error. If you see that error on a name you expected Schema Registry to treat as built-in or local, check it for a typo before adding a schema reference. Register the Logical Type with [schema references](../schema.md#cp-sr-schema-references) to the external types: ```bash curl -X POST -H "Content-Type: application/json" \ --data '{ "schemaType": "AVRO", "schema": "STRUCT Order ( id STRING NOT NULL, quantity INT NOT NULL, ship_to com.example.demo.Address NOT NULL )", "references": [ {"name": "com.example.demo.Address", "subject": "address", "version": 1} ] }' \ 'http://localhost:8081/subjects/orders-value/versions' ``` The simplest reference is a bare reference: ```text TYPE com.example.demo.Address NOT NULL ``` Use `TYPE` here, not `STRUCT`, because `STRUCT` only defines a new user-defined type. This `TYPE` reference points to a `STRUCT` already defined elsewhere. Types are nullable by default here too, so specify `NOT NULL` to get a non-nullable reference. Register it the same way as any other Logical Type, including the `references` array. For Avro, a bare reference resolves to the string `"com.example.demo.Address"`. For Protobuf, it resolves to an `import public` statement, an otherwise-empty file that re-exports the referenced message. Schema Registry resolves through the `import public` when reading the schema back, the same way a Protobuf serializer or deserializer does, so the bare reference works as a value or key schema. For JSON Schema, it resolves to a `$ref` pointing into `$defs`. ## Struct and field clauses Structs and fields can carry the following clauses. `NOT NULL` also applies to any other type position: standalone `TYPE` declarations, a `UNION` and each of its branches, and collection element, key, and value types. `DEFAULT` applies only to fields. The rest apply to both structs and fields: - `NOT NULL`: marks the value as required, disallowing null - `DEFAULT`: field only, a default value for primitive types - `CHECK`: a struct-level or field-level constraint. Write a field-level `CHECK` after the field’s type. Write a struct-level `CHECK` as its own comma-separated entry inside the struct’s parentheses, not after them with `COMMENT`, `TAGS`, and `WITH`. Optionally follow either form with `MESSAGE ''` to set the error message returned when the constraint fails - `COMMENT`: a human-readable description of a struct or field - `TAGS`: labels used to select structs or fields for rule targeting, such as `PII` - `WITH`: key-value parameters for struct- or field-level metadata `DEFAULT` becomes the native `default` field for Avro and JSON Schema. proto3 has no field defaults, so for Protobuf, `DEFAULT` becomes a `connect.default` parameter under `(confluent.field_meta)`, letting `?format=LOGICAL` restore it. This parameter is metadata only. A Protobuf deserializer still reads an unset field as the type’s zero value, such as `0` for `INT`, not as the declared default. For Avro, a nullable field with no `DEFAULT` clause gets no `default` at all, not an implicit `null` default, as the preceding examples show. Adding such a field in a later version breaks `BACKWARD` compatibility, because Avro schema resolution has nothing to fill in for old data. When you add a nullable field in a later version, give it `DEFAULT NULL`. `COMMENT` becomes the native doc field or comment for the target format. For Avro and JSON Schema, `TAGS` becomes a `confluent:tags` property and `WITH` parameters become a `confluent:params` property. For Protobuf, both become `.tags` and `.params` sub-fields of the `(confluent.field_meta)` option on a field, or of the `(confluent.message_meta)` option on a struct. Format-native keys are the exception: any key prefixed `avro.` or `protobuf.`, such as `avro.aliases`. For the format they name, they map to a native attribute, as in the `ssn` field’s alias in the following example. For every other format, Schema Registry drops them. They never appear as an ordinary parameter. Schema Registry also reserves any key prefixed `connect.`, `flink.`, or `logical.`, such as `connect.default`, for the same reason but a different mechanism: it populates them itself to round-trip a value it already stores structurally, such as a field’s default. Setting one directly in `WITH` isn’t supported. `CHECK` compiles to a CEL expression and becomes a [domain rule](data-contracts.md#sr-contracts-rules) in the schema’s `ruleSet`. Schema Registry doesn’t run this check at registration, and it never sees record data. The client serializer or deserializer enforces the rule when it runs data contract rules. A client that ignores `ruleSet`, such as an older serializer or a non-Confluent client, writes data that violates the constraint with no error. Because `CHECK` produces a schema rule, it’s subject to the same requirements as any other rule. Schema Registry must run on Confluent Enterprise with `resource.extension.class=io.confluent.kafka.schemaregistry.rulehandler.RuleSetResourceExtension` set. Clients run rules only for the root schema, not for referenced schemas. For example, the `CHECK` constraints on `Person` in the following example don’t run when you serialize the `Employee` type shown in the next section, which references `Person`. Flink SQL doesn’t run rules. A field-level `CHECK`’s compiled rule skips evaluation entirely when that field’s value is null or absent, so a null value always passes, the same outcome as a SQL `CHECK` on a `NULL` column. The `ssn` and `age` checks in the following example never run against a null value, so pair a field-level `CHECK` with a struct-level one, such as `age IS NOT NULL OR ssn IS NOT NULL`, to require at least one of them to be present. ```text NAMESPACE com.example.common; STRUCT Person ( id BIGINT NOT NULL COMMENT 'Primary key', ssn STRING CHECK (MATCHES(ssn, '^[0-9]{3}-[0-9]{2}-[0-9]{4}$')) MESSAGE 'ssn must look like 123-45-6789' COMMENT 'US social security number' TAGS ('PII') WITH ('avro.aliases' = 'social_security_number'), email STRING COMMENT 'Contact address' TAGS ('PII', 'CONTACT'), age INT CHECK (age BETWEEN 0 AND 150), CHECK (age IS NOT NULL OR ssn IS NOT NULL) MESSAGE 'need age or ssn' ) COMMENT 'A person record' TAGS ('INTERNAL') WITH ('owner' = 'hr-team', 'retention.days' = '365'); ``` ## Local aliases with USING A `USING` directive declares a local alias for a type name. A short alias improves readability when a user-defined type isn’t in the current namespace: ```text NAMESPACE com.example.hr; USING TYPE Person FOR com.example.common.Person; STRUCT Employee ( id BIGINT NOT NULL, person Person ); ``` `Person` isn’t built-in or locally defined as a `STRUCT` or `ENUM` in this script, so it’s still an external reference. `USING` only shortens the name used in the syntax. You still add `com.example.common.Person` to the `references` array when you register this schema. ## Logical Type mappings Avro also has its own, unrelated `logicalType` schema attribute, which the following table uses to encode some Logical Types. The table shows how built-in Logical Types map to Avro, Protobuf, and JSON Schema. | Logical Type | Avro | Protobuf | JSON Schema | |---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------| | `BOOLEAN` | `boolean` | `bool` | `boolean` | | `TINYINT`, `SMALLINT`, and `INT` | `int`. `TINYINT` and `SMALLINT` also set `connect.type`
to `int8` or `int16`, so `?format=LOGICAL` can restore
the original type | `int32` | `number` | | `BIGINT` | `long` | `int64` | `number` | | `FLOAT` | `float` | `float` | `number` | | `DOUBLE` | `double` | `double` | `number` | | `DECIMAL(p,s)` | `bytes`, using `logicalType=decimal`, `precision`, and `scale` | `confluent.type.Decimal` | `number` | | `CHAR(n)`, `VARCHAR(n)`, and `STRING` | `string`. `CHAR(n)` also sets `flink.minLength`, and
both `CHAR(n)` and `VARCHAR(n)` set `flink.maxLength`,
so `?format=LOGICAL` can restore the original length | `string`. Sets the same `flink.minLength` and
`flink.maxLength` parameters as Avro, as sub-fields of
`(confluent.field_meta)` | `string`. `CHAR(n)` sets native `minLength` and
`maxLength`. `VARCHAR(n)` sets only `maxLength` | | `BINARY(n)` | `fixed`, using `size=n` | `bytes` | `string` | | `VARBINARY(n)` and `BYTES` | `bytes` | `bytes` | `string` | | `DATE` | `int`, using `logicalType=date` | `google.type.Date` | `number` | | `TIME(n)` | `int` using `logicalType=time-millis` for `n` 0-3, or
`long` using `time-micros` for `n` 4-9. Avro has no
nanosecond time type, so `n` 7-9 loses precision beyond
microseconds, unlike `TIMESTAMP` | `google.type.TimeOfDay` | `number` | | `TIMESTAMP(n)` | `long`, using `logicalType=local-timestamp-millis` for `n`
0-3, `local-timestamp-micros` for 4-6, or
`local-timestamp-nanos` for 7-9 | `google.protobuf.Timestamp` | `number` | | `TIMESTAMP_LTZ(n)` | `long`, using `logicalType=timestamp-millis` for `n` 0-3,
`timestamp-micros` for 4-6, or `timestamp-nanos` for 7-9 | `google.protobuf.Timestamp` | `number` | | `ARRAY` | `array`, using `items` | `repeated message` when `T` is nullable,
a collection, or a `UNION`. Otherwise, `repeated T`. When
the field itself is nullable, ` message`
wraps the whole `repeated` field | `array`, using `items` | | `MULTISET` | Array of `Entry` record, using `logicalType=map` and
`flink.type=multiset`. Entry fields are `key` and `value`,
the value holding each element’s count | `repeated message` | Array of key-value objects | | `MAP` | `map`, using `values`, when `K` is `CHAR`,
`VARCHAR`, or `STRING` (`STRING` is `VARCHAR` with no
length limit). Otherwise, an array of `Entry` records with
`key` and `value` fields | `repeated message`. When the field itself is nullable,
` message` wraps the whole `repeated` field | `object`, using `additionalProperties`, when `K` is
`CHAR`, `VARCHAR`, or `STRING`. Otherwise, an array of
key-value objects | | `ROW (...)` and inline `STRUCT (...)` | `record` with a generated name, for example `T_f`, tagged
`logical.anonymous` | Nested message, for example `.T.FRow` | `object`, using `properties` and `title` | | `UNION (n T, ...)` | Avro union, for example `["int","string"]` | `oneof` | `oneOf` plus `confluent:union` branch names | | `VARIANT` | `confluent.type.Variant` record, using `logicalType=variant` | `confluent.type.Variant` | `{}` | | `STRUCT Name (...)` | Named record | `.Name` message | Top-level `object` with `title: Name` for the root struct.
Otherwise, `$ref: #/$defs/Name` | | `ENUM Name (...)` | `enum`, using `symbols` | `.Name` enum. Protobuf has no message-less root, so an
`ENUM` can’t be the root. Register it as a field or branch
of a `STRUCT` instead | Top-level schema using `enum` with `title: Name` for the
root `ENUM`. Otherwise, `$ref: #/$defs/Name` | The JSON Schema column uses `number` for every integer and temporal Logical Type, including `TINYINT` through `BIGINT`, `DATE`, `TIME(n)`, and the `TIMESTAMP` types. JSON Schema validation against `number` doesn’t reject a fractional value, unlike the matching Avro and Protobuf forms. None of the three formats rejects a `TINYINT` or `SMALLINT` value outside that type’s range, because Avro and Protobuf store both as a 32-bit integer. The generated name for an inline `ROW` or `STRUCT`, shown in the preceding table as `T_f` for Avro and `.T.FRow` for Protobuf, combines its enclosing type’s name and its own field name in both formats. Renaming either one changes this generated name. Because Avro and Protobuf compatibility checks match nested records or messages by name, even a rename that’s normally safe for the field itself, such as an Avro field with `avro.aliases` or a Protobuf field that keeps its tag, can break compatibility for the nested type. To keep the nested name stable, declare the nested type as a named `STRUCT` instead of an inline `ROW` or `STRUCT`. JSON Schema isn’t affected, because its `title` isn’t used for compatibility matching. For Protobuf, an `ARRAY` element gets the `ElementWrapper` shown in the mapping table when `T` is nullable, which it is by default, or when `T` is a nested collection or `UNION`, whether or not it’s nullable. Write `ARRAY` to keep a plain `repeated string` on the wire. Avro and JSON Schema need no wrapper either way: a nullable element becomes a union or `oneOf` inside `items`, the same as any other nullable field. An `ARRAY` field’s own nullability is different: proto3 forbids `optional repeated`, so a nullable `ARRAY`, `MAP`, or `MULTISET` field wraps the whole `repeated` value in a ` message` instead, shown in the mapping table. Presence on that wrapper message distinguishes a null array from an empty one, because a message field can be unset independent of the `repeated` value inside it. Write `ARRAY NOT NULL` to keep a plain `repeated` field with no wrapper. Avro and JSON Schema need no wrapper either way: a null array is already a distinct value from an empty one in both formats. `MULTISET` and `MAP` don’t use `ElementWrapper` for their element or value type, because their `Entry` message’s `key` and `value` fields are ordinary struct fields, not array elements. A nullable scalar `V` becomes an `optional` field on `Entry`. A collection or `UNION` as `V` triggers the same `RepeatedWrapper` or `OneofWrapper` a nullable field of that shape gets anywhere else, because `Entry`’s fields go through the same field-conversion rules as any other struct’s. For `MULTISET`, the element `T` is the `Entry` message’s `key` field and follows the same rule, and `value` holds the count. For the native `map` and `additionalProperties` forms, a `MAP` key’s own `NOT NULL` clause has no effect on which form it takes. These forms require a plain `CHAR`, `VARCHAR`, or `STRING` key type, regardless of `NOT NULL`, so `MAP` and `MAP` produce the same native form. Keys in these forms can’t be null, so a record with a null key fails at serialization.