Looking for Confluent Platform Cluster Linking docs? This page describes Cluster Linking on Confluent Cloud. If you are looking for Confluent Platform documentation, check out Cluster Linking on Confluent Platform.
Mirror topics are the building blocks for moving data with Cluster Linking. They are read-only topics that are created and owned by a cluster link.
The sections below provide a conceptual overview of mirror topics, how they are created, configured, and how they work in operation, that is applicable to both Confluent Platform and Confluent Cloud.
A cluster link connects a mirror topic to its source topic. Any messages produced to the source topic are mirrored over the cluster link to the mirror topic.
A mirror topic syncs many of its configurations from its source topic. It can also sync ACLs and consumer group offsets from its source topic, if you enable those features on the cluster link.
You can convert a mirror topic to a regular topic using the Cluster Linking
The diagram below shows how mirror topics work, including the relationship between the mirror topic and its source topic, and the syncing of ACLs and consumer offsets.
Mirror topics have these unique properties:
- Mirror topics are created by and owned by a cluster link.
- Mirror topics get their messages from their source topic. They are byte-for-byte, offset-preserving asynchronous copies of their source topics.
- Mirror topics are read-only; you can consume them the same as any other topic, but you cannot produce into them. If a producer tries to produce a message into a mirror topic, the action will fail. The only way to get a message into a mirror topic is to produce the message to the mirror topic’s source topic.
- Many of the mirror topic’s configurations are copied and synced from the source topic. A full list is at the end of this page.
Mirror Topic Creation¶
When you create a mirror topic, you must specify a cluster (as normal) and a cluster link.
This cluster link must have this cluster as its destination cluster. A mirror topic is always created on its cluster link’s destination cluster.
When you create a mirror topic, you must specify a name for it that matches its source topic. Specifically:
There must be a topic of this same name on the cluster link’s source cluster, which will be the mirror topic’s source topic. Mirror topics enforce the convention of preserving topic names. The best practice for a global mesh of clusters is for a topic’s name to reflect the cluster where it originated.
For example, if your
clicksdata is produced to clusters in the US and the EU, your topics could be named
clicks.eu. When you mirror these topics to either cluster, the mirror topic names will reflect where the data came from. To get a full set of
clicksdata, your consumers can consume from the pattern
There must be no topic by this name already on the destination cluster. Mirror topics are always created as brand new topics, so that they can be byte-for-byte copies of their source topic.
The cluster link’s security credentials must be authorized with ACLs to read the source topic.
To create a mirror topic, all of these conditions must be true:
- There must be no topic with that name on the destination cluster (the cluster where the mirror topic is created). Mirror topics are always created as new topics. That is to say, you cannot convert an existing read/write topic into a mirror topic.
- There must be exactly one specified cluster link. not 0, not 2, etc.
- The link’s
- The link’s
dest.cluster.idmust be the mirror topic’s cluster.
- The link’s
bootstrap.serversmust be the mirror topic cluster’s bootstrap servers.
- The link’s
- The link must be in “active” status.
- The destination cluster’s brokers must be able to reach the source cluster’s brokers and verify that there is a suitable source topic:
- A topic on the source cluster with the same name as the mirror topic.
- The security credentials stored on this link must be authorized with the ACLs
DESCRIBE_CONFIGSon the source topic.
- If your destination cluster cannot reach your source cluster with the properties configured on the cluster link, then you will not be able to create a mirror topic.
- The user creating the mirror topic must have these ACLs on the destination cluster:
For more about configuring security, see Cluster Linking Security
This diagram shows an example of a cluster link and a mirror topic configured with some of the properties discussed above.
The mirror process is asynchronous in operation. Therefore, there will often be some mirroring lag between the source topic and the mirror topic. The most recent messages on the source topic may not yet have been mirrored to the mirror topic, so the mirror topic may often be slightly behind the source topic.
The same is true for syncing the topic configuration, the consumer group offsets, and the ACLs. All of these processes are asynchronous, so the changes will happen first on the source topic, and then on the mirror topic shortly after.
Syncing consumer group offsets¶
You can configure your cluster link to sync consumer group offsets. You must pass in a JSON file with a pattern that is matched against consumer group names in order to select which consumer groups to mirror. If these two properties are set, the cluster link syncs the consumer group offsets of any matching consumer groups for all mirror topics that the link mirrors. So, a mirror topic can have consumer group offsets that are synced from the source topic.
To learn more, see
consumer.offset.group.filters in the Confluent Cloud documentation
under Configuring Cluster Link Behavior
and in the Confluent Platform documentation under configuration options
for cluster links.
If consumer group offset syncing is enabled for a given consumer group name, you should not consume from the mirror topic on the destination cluster with that consumer group name, regardless of whether a consumer group of that name is consuming from the source topic. There is no guaranteed behavior when consuming from a mirror topic with a consumer group name that matches the cluster link’s enabled consumer group name filters.
Converting a Mirror Topic to a Normal Topic¶
If you want to convert a mirror topic into a normal topic that you can produce
into, you can call the
failover or the
promote command on the mirror topic.
In Confluent Platform, you call
kafka-mirrors --promote or
In Confluent Cloud, you call
ccloud kafka mirror promote <topic-name> or
ccloud kafka mirror failover <topic-name>.
Both commands occur on the destination cluster (the mirror topic’s cluster) and require you to pass in the cluster link’s name.
promote option is often used for migrations. It checks that there is no mirroring lag, config sync lag, or
consumer offset lag between the source topic and the mirror topic. Then, it converts the mirror topic into a full topic,
with the assurance that this topic was exactly the same as its source topic.
The destination cluster’s brokers must be able to reach the source cluster’s brokers in order to make this check, so your source cluster must be online.
failover option is often used when a disaster has hit the source cluster (for example, a cloud region outage)
and you want to shift operations from the source topic to the mirror topic. This command will succeed regardless of the
mirroring lag or the source cluster’s reachability.
failover commands are irreversible. There is no way to change this topic back into a mirror topic.
If you want a mirror topic of the same name as the one you promoted or failed over on this cluster, you must delete the converted topic, and create a new mirror topic of the same name.
There is no way to change a mirror topic to use a different cluster link or make changes to the link itself, other than to recreate the mirror topic on a different link.
You can run
mirror describe (
ccloud kafka mirror describe <mirror-topic-name> --link <link>) on a promoted or
failed over mirror topic, if you do not delete the cluster link. If you delete the cluster link, you will lose the history and,
therefore, mirror describe will not find data on promoted or failed over topics.
Mirror Topic States and Statuses¶
When you describe a mirror topic, it will return one of these states:
- The mirror is running normally, and messages are being mirrored from the source topic to the destination topic.
- A user has paused mirroring for this mirror topic.
- To reach this state, a user must either pause this specific topic, or pause its cluster link.
- A user has stopped this mirror topic with the
promotecommand, and this topic will soon be in the
- To force the mirror topic to immediately go from the
PENDING_STOPPEDstate to the
STOPPEDstate, call the
failovercommand on it. Doing this cancels any synchronization that was happening between the source cluster and the destination cluster, and eliminates any guarantees that the
- A user has stopped this mirror topic with the
- Mirroring has permanently stopped for this topic. It will no longer receive messages from its source topic. The topic is now writable and can receive messages produced directly to it.
- To get into this state, a user must call either
failoveron this mirror topic.
- Even though a
STOPPEDtopic is no longer a mirror topic, it will still be listed in output for the commands
ccloud kafka mirror listand
ccloud kafka mirror describe <destination-topic-name> --link <link>for as long as the cluster link exists. This is useful because the topic will return the last offset it fetched from its source topic (Last Source Fetch Offset) for each partition, and the time at which it was stopped (Status Time).
- The mirror topic is unable to reach the source topic, and is not mirroring messages from the source topic. This could happen if the source cluster is experiencing an outage or if the network between the destination cluster and the source cluster is unstable.
- Mirroring will resume once the issue is resolved and the destination cluster can reach the source cluster.
- An error has broken the mirror topic’s cluster link, and no data is being mirrored. A user needs to manually re-configure the link.
- The mirror topic has permanently failed. It will no longer mirror data. This can happen if the cluster link ACLs are removed from the source cluster, or if the source topic is deleted. In both cases, the failed status takes effect only after cluster.link.retry.timeout.ms is reached (by default, the system retries the link for 5 minutes).
- You can stop this mirror with the
failovercommand, and it will become a regular topic.
- If you want to restore mirroring for this topic, you must delete the legacy mirror topic and create a new mirror topic with the same name.
Mirror Topic Deletion¶
You may safely delete a mirror topic. Deleting a mirror topic permanently stops data mirroring to that topic. If you create a new normal topic of the same name on the same cluster, data will not be mirrored to it.
You cannot delete a cluster link that is attached to any mirror topics. You must first delete, failover, or promote all of the mirror topics, and then you can delete the cluster link.
Source Topic Deletion¶
Do not delete a source topic that is being mirrored by a mirror topic. Doing so can lead to unpredictable truncation and data loss on the mirror topic. You should always stop mirroring to all associated mirror topics before deleting a source topic.
While it may be possible to delete a source topic that is being mirrored by a mirror topic and a cluster link, doing so is not recommended. In particular, unpredictable behavior can occur if a source topic is deleted, and a topic by the same name is then created within a few minutes time. This scenario can cause permanent data loss on any mirror topics that are still mirroring from that source topic, and can also cause performance issues on the source cluster or destination cluster.
Before deleting a source topic, you should stop any mirroring to associated mirror topics. You can stop mirroring on a mirror topic in one of these ways:
- Delete the mirror topic.
failoverso the mirror topic enters the
- Revoke the security permissions for the cluster link to read the mirror topic.
You can do this in one of three ways: (1) delete the cluster link’s
ALLOWACL for the source topic, (2) create a
DENYACL for the source topic, or (3) delete the cluster link’s API key.
For further discussion about Kafka’s limitations with topic deletion and how topic IDs will help, see KIP-516.
Advanced Mirror Topic Architectures¶
A mirror topic can be a source topic itself. A mirror topic can be mirrored by different cluster links, allowing you to “chain” cluster links and mirror topics together.
For example, Topic A (source topic) on Cluster 1 —cluster link—> Topic A (mirror topic and source topic) on Cluster 2 —cluster link—> Topic A (mirror topic) on Cluster 3
You can safely create these chained topics and cluster links without creating a circular dependency between mirror topics and cluster links. You can create mirror topics without the fear of creating an infinite loop.
A source topic can be mirrored to multiple mirror topics. These mirror topics must exist on multiple different clusters.
For example, Topic A on Cluster 1 —cluster link—> Topic A on Cluster 4, and Topic A on Cluster 1 —cluster link—> Topic A on Cluster 5
If you plan to use
promote on a cluster link (for example, for Disaster Recovery or Migration), then chained or
fanned-out mirror topics will not automatically retain their shape. For example, if you fan out A –> B and A –> C, if A has an outage
and you call failover on B, there is no way to automatically mirror B –> C. You will need to reconstruct the appropriate mirroring
relationship for your use case using brand new topics.
The following sections provide a quick reference of which Cluster Linking configurations are synced from the source to the mirror topic, overrides, and concepts related to syncing.
Synced Mirror Topic Configurations for Confluent Cloud¶
These configurations are always synced from the source topic to the mirror topic. Mirror topics will always have the same value as their source topic, in order to ensure the properties of mirror topics are met.
- number of partitions
Mirror Topic Configurations Not Synced¶
Any configuration that is not in the list above will not be synced to a mirror topic in Confluent Cloud. Therefore, the mirror topic’s configuration could be different from the source topic’s configuration. If you don’t override the mirror topic’s configuration, then it will inherit its cluster’s default.
A few important examples of configurations that are not synced to mirror topics in Confluent Cloud:
replication.factor(No replication factors are synced to mirror topics. The replication factor defaults to
3for all topics, and this is not configurable.)
Hybrid Cloud Configuration Syncs¶
Confluent Platform and Confluent Cloud have different policies for which mirror topic configurations are synced. If you cluster link between Confluent Platform and Confluent Cloud, then the destination cluster’s policy is enforced.
For example, if you cluster link from a Confluent Platform source cluster to a Confluent Cloud destination cluster,
then the value of
compression.type will not be synced. But if you cluster link from a Confluent Cloud source cluster
to a Confluent Platform destination cluster, then
compression.type will be synced.