Lightning Tables Support and Limitations in Confluent Cloud

Lightning Tables support Basic, Standard, Enterprise, and Dedicated clusters on AWS, for topics with Avro, Protobuf, or JSON Schema data.

For supported data types and SQL statements, see SQL Reference for Lightning Table Queries in Confluent Cloud.

Supported environments

Lightning Tables are available on the following cluster types on AWS:

Basic
Standard
Enterprise
Dedicated

For the list of supported regions, see Cloud Regions.

Data model requirements

To enable a Lightning Table for a topic, the topic must have a registered schema in Avro, Protobuf, or JSON Schema format.

Upsert mode

A Lightning Table automatically selects the ingestion mode based on the topic’s cleanup.policy configuration. In upsert mode, the Lightning Table deduplicates records by key so that it reflects the latest state for each key. This behavior is consistent with Apache Kafka® log compaction.

Upsert requirements

To use upsert mode, your topic must meet the following requirements in addition to the general data model requirements:

The topic’s cleanup.policy must include compact. Topics with the default delete policy use append mode.
The topic must have a Kafka record key, which the Lightning Table uses as the primary key for deduplication. The key can be raw bytes, or it can use a schema in Avro, Protobuf, or JSON Schema format. A Lightning Table doesn’t support changes to the key schema. For more information, see Schema Evolution for Lightning Tables in Confluent Cloud.

Upsert behavior

Upsert mode has the following behavior:

Last-write-wins: When multiple records share the same key, the Lightning Table keeps the most recent record and replaces earlier records with the same key.
Deletes: A record with a null value, known as a tombstone, deletes its key from the Lightning Table. A later record with the same key adds it back.