Real-Time Context Engine Support and Limitations in Confluent Cloud

The Real-Time Context Engine supports Basic, Standard, Enterprise, and Dedicated clusters on AWS, using Avro, Protobuf, or JSON Schema data. Some data types and query patterns have restrictions.

Supported environments

The Real-Time Context Engine is available on the following cluster types on AWS:

Basic
Standard
Enterprise
Dedicated

For the current list of supported regions, see Cloud Regions.

Supported data types

You can enable the Real-Time Context Engine on topics that use the following data types in their schema. The Real-Time Context Engine supports Avro, Protobuf, and JSON Schema formats.

Numeric types

The following numeric types are supported:

BOOLEAN
TINYINT
SMALLINT
INT
BIGINT
FLOAT
DOUBLE
DECIMAL

String and binary types

The following string and binary types are supported:

CHAR
VARCHAR
BINARY
VARBINARY

Temporal types

The following temporal types are supported:

DATE
TIME (precision 0 only)
TIMESTAMP (precision 0-6)
TIMESTAMP_LTZ (precision 0-6)

Note

TIMESTAMP and TIMESTAMP_LTZ with precision 7-9 are not supported in Avro or JSON Schema formats.

Complex types

The following complex types are supported:

ARRAY
MAP — Note that MAP types with nullable non-character keys are not supported.
ROW (also known as STRUCT)

Unsupported data types

The Real-Time Context Engine does not support the following data types:

INTERVAL YEAR TO MONTH
INTERVAL DAY TO SECOND
MULTISET
TIMESTAMP WITH TIME ZONE
RAW

If your topic schema uses an unsupported data type, you cannot enable the Real-Time Context Engine on that topic.

Data model requirements

To enable the Real-Time Context Engine on a topic, the topic must have a registered schema in Avro, Protobuf, or JSON Schema format.

Upsert mode

The Real-Time Context Engine automatically selects the ingestion mode based on the topic’s cleanup.policy configuration. In upsert mode, the Real-Time Context Engine deduplicates records by key so that the materialized table reflects the latest state for each key, consistent with Kafka log compaction.

Upsert requirements

To use upsert mode, your topic must meet the following requirements in addition to the general data model requirements:

The topic’s cleanup.policy must include compact. Topics with the default delete policy use append mode.
The topic must have a Kafka record key, which the Real-Time Context Engine uses as the primary key for deduplication. The key must use raw byte format; structured key schemas (Avro, Protobuf) are not supported.

Upsert behavior

Upsert mode has the following behavior:

Last-write-wins: When multiple records share the same key, the most recent record is kept. Earlier records with the same key are replaced.
Deletes: A record with a null value, known as a tombstone, deletes its key from the materialized table. A later record with the same key adds it back.

Query limitations

The Real-Time Context Engine supports read-only queries only. The following operations are not supported:

Aggregates (COUNT, SUM, AVG, MIN, MAX)
GROUP BY
Joins
Subqueries
Set operations (UNION, EXCEPT, INTERSECT)
DDL statements (CREATE, ALTER, DROP)
DML statements (INSERT, UPDATE, DELETE)

For query limits including maximum rows, timeout, and rate limits, see Query limits.

Schema compatibility and evolution

The Real-Time Context Engine supports schema evolution by using Confluent Schema Registry as the source of truth. When evolving event schemas for Apache Kafka® topics, schemas must adhere to the defined compatibility rules.

Materialization is the continuous process that keeps a topic’s data available to query through the Real-Time Context Engine. During materialization, the Real-Time Context Engine validates schema changes against Schema Registry to ensure compliance. If the Real-Time Context Engine encounters a schema change that doesn’t conform to the supported evolution rules, it suspends materialization for the affected topic.

Supported schema evolutions

The Real-Time Context Engine supports backward-compatible evolution that allows the following schema changes:

Adding a new optional column that has a default value of null to ensure backward compatibility with data produced by older schema versions.
Widening a type:
- int to long
- float to double
Relaxing nullability, which means changing a required field to optional.

The Real-Time Context Engine doesn’t rewrite existing data. New and old data coexist and are queryable together.

Schema limitations

The following schema changes are not supported and cause the Real-Time Context Engine to suspend materialization for the topic:

Changing the schema of a key.
Dropping columns.
Renaming columns.
Reordering columns.
Adding a required (non-nullable) column.
Narrowing a type, for example, long to int.
Changing a type incompatibly, for example, string to int.
Tightening nullability, which means changing an optional field to required.
Adding new nested fields to an existing schema.

Migrate to a new schema

Migrate to a new topic with an updated schema by using Apache Flink® to transform and copy your data. Use this approach when you need to make an incompatible schema change.

To migrate to a new topic, complete the following steps:

Create a new Kafka topic and register the updated schema in Schema Registry. The new schema must meet all Real-Time Context Engine requirements.
Start a continuous Flink SQL job that reads from the original topic, applies the necessary transformations (for example, column renames, type casts, column removal), and writes to the new topic. Start this job before enabling the Real-Time Context Engine on the new topic so that data is already flowing when materialization begins.
Enable the Real-Time Context Engine on the new topic.
Verify that the new topic is serving the expected data and update any agent configurations that reference the original topic name.
Disable the Real-Time Context Engine on the original topic.

Note

The Real-Time Context Engine materializes both topics during the transition window. Kafka retention on the original topic must be sufficient for the Flink job to replay the full history.

If you need help with an incompatible schema change, contact Confluent Support.

Private networking

The Real-Time Context Engine supports private networking on AWS.

Warning

Cross-environment and cross-network access

With private networking enabled, topics enabled with the Real-Time Context Engine might be accessible across environments and private network boundaries within your organization. This is a temporary limitation. Review your private network topology and environment isolation model before enabling the feature.

You can connect to the Real-Time Context Engine endpoint by using the following networking options:

The following table shows the endpoint patterns for different networking and cluster type combinations.

Networking	DNS	Cluster Type	Endpoint
PrivateLink	Private	Enterprise (PrivateLink Gateway)	`mcp.$region.$cloud.private.confluent.cloud`
PrivateLink	Private	Dedicated	`mcp.dom$id.$region.$cloud.private.confluent.cloud`
PrivateLink	Public	Dedicated	`mcp-$nid.$region.$cloud.glb.confluent.cloud`
VPC Peering / Transit Gateway w/ /16 CIDR	Public	Dedicated	`mcp-$nid.$region.$cloud.confluent.cloud`
VPC Peering / Transit Gateway w/ /27 CIDRs	Public	Dedicated	`mcp-$nid.$region.$cloud.glb.confluent.cloud`