# Real-Time Context Engine Support and Limitations in Confluent Cloud The Real-Time Context Engine supports Basic, Standard, Enterprise, and Dedicated clusters on AWS, using Avro, Protobuf, or JSON Schema data. Some data types and query patterns have restrictions. ## Supported environments The Real-Time Context Engine is available on the following cluster types on AWS: - Basic - Standard - Enterprise - Dedicated For the current list of supported regions, see [Cloud Regions](../../get-started/regions.md#providers-regions). ## Supported data types You can enable the Real-Time Context Engine on topics that use the following data types in their schema. The Real-Time Context Engine supports Avro, Protobuf, and JSON Schema formats. ### Numeric types The following numeric types are supported: - `BOOLEAN` - `TINYINT` - `SMALLINT` - `INT` - `BIGINT` - `FLOAT` - `DOUBLE` - `DECIMAL` ### String and binary types The following string and binary types are supported: - `CHAR` - `VARCHAR` - `BINARY` - `VARBINARY` ### Temporal types The following temporal types are supported: - `DATE` - `TIME` (precision 0 only) - `TIMESTAMP` (precision 0-6) - `TIMESTAMP_LTZ` (precision 0-6) #### NOTE `TIMESTAMP` and `TIMESTAMP_LTZ` with precision 7-9 are not supported in Avro or JSON Schema formats. ### Complex types The following complex types are supported: - `ARRAY` - `MAP` — Note that `MAP` types with nullable non-character keys are not supported. - `ROW` (also known as `STRUCT`) ## Unsupported data types The Real-Time Context Engine does not support the following data types: - `INTERVAL YEAR TO MONTH` - `INTERVAL DAY TO SECOND` - `MULTISET` - `TIMESTAMP WITH TIME ZONE` - `RAW` If your topic schema uses an unsupported data type, you cannot enable the Real-Time Context Engine on that topic. ## Data model requirements To enable the Real-Time Context Engine on a topic, the topic must have a registered schema in Avro, Protobuf, or JSON Schema format. ## Upsert mode The Real-Time Context Engine automatically selects the ingestion mode based on the topic’s [cleanup.policy](../../topics/manage.md#topics-cleanup-policy) configuration. In upsert mode, the Real-Time Context Engine deduplicates records by key so that the materialized table reflects the latest state for each key, consistent with Kafka log compaction. ### Upsert requirements To use upsert mode, your topic must meet the following requirements in addition to the [general data model requirements](#rtce-data-model-requirements): - The topic’s `cleanup.policy` must include `compact`. Topics with the default `delete` policy use append mode. - The topic must have a Kafka record key, which the Real-Time Context Engine uses as the primary key for deduplication. The key must use raw byte format; structured key schemas (Avro, Protobuf) are not supported. ### Upsert behavior Upsert mode has the following behavior: - **Last-write-wins**: When multiple records share the same key, the most recent record is kept. Earlier records with the same key are replaced. - **Deletes**: A record with a null value, known as a tombstone, deletes its key from the materialized table. A later record with the same key adds it back. ## Query limitations The Real-Time Context Engine supports read-only queries only. The following operations are not supported: - Aggregates (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`) - `GROUP BY` - Joins - Subqueries - Set operations (`UNION`, `EXCEPT`, `INTERSECT`) - DDL statements (`CREATE`, `ALTER`, `DROP`) - DML statements (`INSERT`, `UPDATE`, `DELETE`) For query limits including maximum rows, timeout, and rate limits, see [Query limits](query-data.md#rtce-query-limits). ## Schema compatibility and evolution The Real-Time Context Engine supports schema evolution by using Confluent Schema Registry as the source of truth. When evolving event schemas for Apache Kafka® topics, schemas must adhere to the defined compatibility rules. Materialization is the continuous process that keeps a topic’s data available to query through the Real-Time Context Engine. During materialization, the Real-Time Context Engine validates schema changes against Schema Registry to ensure compliance. If the Real-Time Context Engine encounters a schema change that doesn’t conform to the supported evolution rules, it suspends materialization for the affected topic. ### Supported schema evolutions The Real-Time Context Engine supports backward-compatible evolution that allows the following schema changes: - Adding a new optional column that has a default value of `null` to ensure backward compatibility with data produced by older schema versions. - Widening a type: - `int` to `long` - `float` to `double` - Relaxing nullability, which means changing a required field to optional. The Real-Time Context Engine doesn’t rewrite existing data. New and old data coexist and are queryable together. ### Schema limitations The following schema changes are not supported and cause the Real-Time Context Engine to suspend materialization for the topic: - Changing the schema of a key. - Dropping columns. - Renaming columns. - Reordering columns. - Adding a required (non-nullable) column. - Narrowing a type, for example, `long` to `int`. - Changing a type incompatibly, for example, `string` to `int`. - Tightening nullability, which means changing an optional field to required. - Adding new nested fields to an existing schema. ### Migrate to a new schema Migrate to a new topic with an updated schema by using Apache Flink® to transform and copy your data. Use this approach when you need to make an incompatible schema change. To migrate to a new topic, complete the following steps: 1. Create a new Kafka topic and register the updated schema in Schema Registry. The new schema must meet all Real-Time Context Engine requirements. 2. Start a continuous Flink SQL job that reads from the original topic, applies the necessary transformations (for example, column renames, type casts, column removal), and writes to the new topic. Start this job before enabling the Real-Time Context Engine on the new topic so that data is already flowing when materialization begins. 3. Enable the Real-Time Context Engine on the new topic. 4. Verify that the new topic is serving the expected data and update any agent configurations that reference the original topic name. 5. Disable the Real-Time Context Engine on the original topic. #### NOTE The Real-Time Context Engine materializes both topics during the transition window. Kafka retention on the original topic must be sufficient for the Flink job to replay the full history. If you need help with an incompatible schema change, contact [Confluent Support](https://support.confluent.io). ## Related content - [Overview](overview.md#real-time-context-engine) - Feature overview and supported cluster types. - [Get Started](get-started.md#rtce-get-started) - Enable the Real-Time Context Engine and run your first query. - [Query Data](query-data.md#rtce-query-data) - Supported query patterns and MCP tools. - [Private Networking](private-networking.md#rtce-private-networking) - Supported networking options and endpoint formats. #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).