Serialize and Deserialize data

Confluent Cloud for Apache Flink®️ supports records in the Avro, JSON_SR, and Protobuf formats.

Important

Confluent Cloud for Apache Flink®️ is currently available for Preview. A Preview feature is a Confluent Cloud component that is being introduced to gain early feedback from developers. Preview features can be used for evaluation and non-production testing purposes or to provide feedback to Confluent. The warranty, SLA, and Support Services provisions of your agreement with Confluent do not apply to Preview features. Confluent may discontinue providing Preview releases of the Preview features at any time in Confluent’s sole discretion. Check out Getting Help for questions, feedback and requests.

For SQL features and limitations in the preview program, see Notable Limitations in Public Preview.

Avro schemas

Known limitations

  • Avro unions are generally not supported. Apache Flink® supports only unions of two elements where one of the elements is NULL. This is how Flink represents a nullable type.
  • Avro enums have limited support. Flink supports reading and writing enums but treats them as a STRING type. From Flink’s perspective, enums are not distinguishable from the STRING type. You can’t create an Avro schema from Flink that has an enum field.
  • Flink doesn’t support reading Avro time-micros as a TIME type. Flink supports TIME with precision up to 3. time-micros are read and written as BIGINT.
  • These Flink types are not supported:
    • INTERVAL_DAY_TIME
    • INTERVAL_YEAR_MONTH
    • TIMESTAMP_WITH_TIMEZONE

JSON schemas

Notes

  • Nullable types are expressed as oneOf(NullSchema, T).
  • ObjectSchema for a MAP and MULTISET must have two fields [key, value].
  • MULTISET is equivalent to MAP[K, INT] and is serialized accordingly.

Notes

  • Nullable types are expressed as oneOf(NullSchema, T).
  • ObjectSchema for a MAP and MULTISET must have two fields [key, value].
  • CombinedSchema (oneOf, allOf, anyOf) is expressed as a ROW, unless it can be simplified, for example, StringSchema and ConstantSchema.

Protobuf schemas

Notes

Expressing something as NULLABLE or NOT NULL is not straightforward in Protobuf.

  • All non-MESSAGE types are NOT NULL. If not set explicitly, the default value is assigned.
  • Non-MESSAGE types marked with “optional” can be checked if they were set. If not set, SQL assumes NULL.
  • MESSAGE types are all NULLABLE, which means that all fields of MESSAGE type are optional, and there is no way to ensure on a format level they are NOT NULL.
  • ARRAYS can’t be NULL, and a not-set repeated field is presented as an empty list. There is no way to differentiate an empty array from NULL.