Mirror Topics

Looking for Confluent Cloud Cluster Linking docs? You are currently viewing Confluent Platform documentation. If you are looking for Confluent Cloud docs, check out Cluster Linking on Confluent Cloud.

What are Mirror Topics?

Mirror topics are the building blocks for moving data with Cluster Linking. They are read-only topics that are created and owned by a cluster link.

The sections below provide a conceptual overview of mirror topics, how they are created, configured, and how they work in operation, that is applicable to both Confluent Platform and Confluent Cloud.

Overview

A cluster link connects a mirror topic to its source topic. Any messages produced to the source topic are mirrored over the cluster link to the mirror topic.

A mirror topic syncs many of its configurations from its source topic. It can also sync ACLs and consumer group offsets from its source topic, if you enable those features on the cluster link.

You can convert a mirror topic to a regular topic using the Cluster Linking promote and failover commands.

The diagram below shows how mirror topics work, including the relationship between the mirror topic and its source topic, and the syncing of ACLs and consumer offsets.

../../_images/cluster-link-mirror-topics-example.png

Mirror Topic Fundamentals

Properties

Mirror topics have these unique properties:

  • Mirror topics are created by and owned by a cluster link.
  • Mirror topics get their messages from their source topic. They are byte-for-byte, offset-preserving asynchronous copies of their source topics.
  • Mirror topics are read-only; you can consume them the same as any other topic, but you cannot produce into them. If a producer tries to produce a message into a mirror topic, the action will fail. The only way to get a message into a mirror topic is to produce the message to the mirror topic’s source topic.
  • Many of the mirror topic’s configurations are copied and synced from the source topic. A full list is at the end of this page.

Mirror Topic Creation

Command syntax

The general syntax used to create a mirror topic is:

kafka-mirrors --create --mirror-topic <topic-name> \
--link <link-name> \
--bootstrap-server <host:port>

You can create a specific mirror topic using the Confluent Cloud Console, the Confluent Cloud REST API, the Confluent CLI, the Confluent Platform AdminClient API, or Confluent for Kubernetes. Alternatively, you can configure your cluster link to automatically create mirror topics that match certain prefixes.

To learn more, see confluent kafka mirror create in the Confluent CLI command reference.

Creating a mirror topic with the AdminClient API

You can use the AdminClient API to create mirror topics on Confluent Platform. To learn more, see ConfluentAdmin API reference.

Examples

For examples of how to create mirror topics on Confluent Platform, see Create the cluster link and mirror topic (step 2, “Initialize the mirror topic”) in the basic tutorial and Creating a mirror topic in the Commands documentation.

For examples of how to create mirror topics on Confluent Cloud, see the following sections:

Requirements

When you create a mirror topic, you must specify a cluster (as normal) and a cluster link.

This cluster link must have this cluster as its destination cluster. A mirror topic is always created on its cluster link’s destination cluster.

Mirror topics enforce the convention of preserving topic names. The best practice for a global mesh of clusters is for a topic’s name to reflect the cluster where it originated.

For example, if your clicks data is produced to clusters in the US and the EU, your topics could be named clicks.us and clicks.eu. When you mirror these topics to either cluster, the mirror topic names will reflect where the data came from. To get a full set of clicks data, your consumers can consume from the pattern clicks.*

If prefixing is enabled on the cluster link, you must specify both the source topic and mirror topic name when you create the mirror topic. (See Additional requirements when prefixing is enabled.)

To create a mirror topic, all of these conditions must be true:

  • There must be no topic with that name on the destination cluster (the cluster where the mirror topic is created). Mirror topics are always created as new topics. That is to say, you cannot convert an existing read/write topic into a mirror topic.
  • There must be exactly one specified cluster link. not 0, not 2, etc.
    • The link’s link.mode must be DESTINATION.
    • The link’s dest.cluster.id must be the mirror topic’s cluster.
    • The link’s bootstrap.servers must be the mirror topic cluster’s bootstrap servers.
  • The link must be in “active” status.
  • The destination cluster’s brokers must be able to reach the source cluster’s brokers and verify that there is a suitable source topic:
    • A topic on the source cluster with the same name as the mirror topic, excluding the cluster link’s prefix (if a prefix is configured on the cluster link). (See also, Additional requirements when prefixing is enabled and Prefixing Mirror Topics and Consumer Group Names.)
    • The security credentials stored on this link must be authorized with the ACLs READ and DESCRIBE_CONFIGS on the source topic.
    • If your destination cluster cannot reach your source cluster with the properties configured on the cluster link, then you will not be able to create a mirror topic.
  • The user creating the mirror topic must have these ACLs on the destination cluster:
    • ALTER the cluster
    • CREATE the topic
    • ALTER the topic

For more about configuring security, see Cluster Linking Security.

Additional requirements when prefixing is enabled

When a cluster link has a prefix set, the specified prefix will be added to the beginning of mirror topic names. For example; if you set the prefix to west, the source topic orders will be mirrored as west.orders.

If the cluster link is configured for prefixing mirror topic names, then to create a mirror topic you must pass both the mirror topic name and the source topic name (instead of just the source topic name).

To learn more about prefixing, see Prefixing Mirror Topics and Consumer Group Names.

Bidirectional cluster linking

To establish bidirectional linking between two clusters, you must use two cluster links. You cannot establish bi-directional linking with a single cluster link. For an example of bidirectional linking, see the Hybrid tutorial (on either Confluent Cloud or Confluent Platform), which sets up bidirectional linking between on-premises and cloud clusters.

Bidirectional linking is supported for different topics. For a specific topic, only unidirectional linking is supported.

Cherry pick which topics to mirror

To cherry pick topics to be mirrored, you can use any of the following methods:

Support for compacted topics

Cluster Linking supports compacted topics. A compacted topic is mirrored as such from source to destination. To learn more, see the FAQs for Confluent Cloud and Confluent Platform.

Auto-Create Mirror Topics

A cluster link is able to automatically create mirror topics on the destination cluster for any topics that exist on the source cluster. This is called “auto-creating” mirror topics. This saves time and effort, because you do not have to create mirror topics by hand. This functionality can be scoped down to a specific set of topics by matching on the topics’ names.

Enabling auto-create mirror topics

To enable this functionality, you must set two properties on the cluster link. You can set these properties when a cluster link is created, or update an existing cluster link with these properties. These properties are:

auto.create.mirror.topics.enable

Whether or not to auto-create mirror topics based on topics on the source cluster. When set to “true”, mirror topics will be auto-created. Setting this option to “false” disables mirror topic creation and clears any existing filters.

  • Type: boolean
  • Default: false
auto.create.mirror.topics.filters
  • A JSON object with one property, topicFilters, that contains an array of filters to apply to indicate which topics should be mirrored. Filters are described below.
  • This list must have at least one filter.
  • Ordering of the filters in this array does not matter.
  • Type: array
  • Default: empty

Syntax

{ “topicFilters”: [ <each filter to apply> ] }

Filters for auto-create mirror topics

In Confluent Cloud, auto-creating mirror topics automatically filters out Confluent internal topics and the topic that holds schemas (default name _schemas).

In Confluent Platform, internal topics may be filtered out in upcoming releases, but in current releases, they are not. All other filtering options described below are available in both Confluent Cloud and current releases of Confluent Platform.

Other topics can be excluded using filters. For example, if a different topic name is used for Schema Registry storage, instead of _schemas, it can be excluded by using filters. Further detail on how to filter topics for auto-create mirror topics is provided below.

You can select exactly which source topics to automatically mirror through a list of filters. There is no limit to the number of filters you can add on a cluster link.

Each filter is a JSON object with the following fields:

name
Text that will be matched against the name of the topic. Set name to the wildcard, *, to apply to all topics.
patternType

Either LITERAL or PREFIXED

  • If name is set to “foo”, then setting patternType to LITERAL will only match a topic named “foo”.
  • Setting patternType to PREFIXED will match any topic names that begin with “foo”, for example, “foo”, “football”, and “foo.fighters”.
filterType

Either INCLUDE or EXCLUDE

  • If filterType is set to INCLUDE, any topic names on the source cluster that match this filter will be created as mirror topics.
  • If filterType is set to EXCLUDE , any matching topic names will not be created as mirror topics. In other words, prevents auto mirror topic creation for the specified topic names. EXCLUDE filters override any overlapping INCLUDE filters. For example, if you have an INCLUDE filter for the prefix “foo” but have an EXCLUDE filter for the prefix “foo.bar,” then a topic on the source cluster named “foo.fighters” would be mirrored automatically, but a topic named “foo.bar.fighters” would not be mirrored automatically.

There is no limit to the number of filters you can add on a cluster link.

Example Filters

Mirror all topics

This filter will create mirror topics for all current and future source cluster topics:

{ "topicFilters": [ {"name": "*",  "patternType": "LITERAL",  "filterType": "INCLUDE"} ] }

Mirror all topics that begin with a given string

This filter will mirror all topics that begin with “foo”:

{ "topicFilters": [ {"name": "foo",  "patternType": "PREFIXED",  "filterType": "INCLUDE"} ] }

Mirror all topics except those that begin with “secret”

This filter will mirror all topics except those that begin with “secret”:

{ "topicFilters": [ {"name": "*",  "patternType": "LITERAL",  "filterType": "INCLUDE"},   \
{"name": "secret",  "patternType": "PREFIXED",  "filterType": "EXCLUDE"} ] }

Mirror named topics if they exist on the source cluster

This filter will mirror three topics, “liz”, “jack”, and “kenneth”, if they exist on the source cluster:

{ "topicFilters": [ {"name": "liz",  "patternType": "LITERAL",  "filterType": "INCLUDE"},   \
{"name": "jack",  "patternType": "LITERAL",  "filterType": "INCLUDE"},    \
{"name": "kenneth",  "patternType": "LITERAL",  "filterType": "INCLUDE"}  ] }

How a mirror topic is auto-created

For a given topic on a cluster link’s source cluster (the “source topic”), a new mirror topic will be auto-created by the cluster link if all of these conditions are true:

  • auto.create.mirror.topics.enable is set to true
  • auto.create.mirror.topics.filters has filters which INCLUDE the source topic name
  • The cluster link’s security credential is authorized (via source cluster ACLs) to read the source topic.
  • There is no topic by that name already on the destination cluster.
  • If prefixing is enabled on the cluster link, then the source topic cannot be a mirror topic. You cannot “chain” mirror topics when both auto.create.mirror.topics.enable and prefixing are enabled.

If any of the above conditions are false, then a mirror topic will not be auto-created for the given source topic.

Deleting topics that were auto-created

You cannot delete a mirror topic that matches the auto-create mirror topics filters. If you deleted such a topic, and there was a topic of the same name on the source cluster, the mirror topic would be automatically re-created and sync all of its history. This would render the delete operation futile.

To delete a mirror topic while auto-create mirror topics is enabled, you have three options: delete the source topic first, exclude the topic’s name from the auto-create mirror topics filters, or disable auto-create mirror topics.

  • Option 1. Delete the source topic first - Given a source topic named cool-topic, if you delete the source topic and then want to subsequently delete the associated mirror topic (cool-topic on the destination), wait until the mirror topic becomes a FAILED mirror topic (which may take up to five minutes), after which point you can delete it. You can also call failover or promote on the mirror topic to transition it to the STOPPED state. Both FAILED and STOPPED mirror topics can be deleted.
  • Option 2. Exclude the topic name from the auto-create mirror topics filters - This strategy will prevent the mirror topic from overlapping with the auto-create filters. An easy way to remove a given topic from the filters is to add an EXCLUDE filter for that topic name. You can add cool-topic to the EXCLUDE filters, even if no such source topic exists. After editing the auto-create mirror topic filters, you can delete the mirror topic.
  • Option 3: Disable auto-create mirror topics on the cluster link - After the setting has been disabled, the mirror topic can be deleted. If needed, auto-create mirror topics can be immediately re-enabled on the cluster link. To learn more, see Disabling auto-create mirror topics and Mirror Topic Deletion.

Disabling auto-create mirror topics

To disable auto-create mirror topics entirely, set this property on the cluster link:

auto.create.mirror.topics.enable=false

Here’s an example of how to set that property with the CLI:

echo "auto.create.mirror.topics.enable=false" > tmp.txt
confluent kafka link update <link-name> --config-file tmp.txt
rm tmp.txt

Prefixing Mirror Topics and Consumer Group Names

Cluster links can be configured with a prefix (cluster.link.prefix) that is applied to the names of the mirror topics and, optionally, the names of the consumer groups that are managed by the cluster link at the destination cluster. This enables topics and consumer groups from different source clusters that have the same name to be synced to the destination without name clashes. It also enables all mirror topics from a cluster link to be categorized and managed under one prefix on the destination.

For example, consider two links, link-1 and link-2. link-1 is linking data from cluster s1 to destination and link-2 is linking data from s2 to destination, and furthermore s1 and s2 both contain a topic “clicks”. Without prefixing, it would be impossible for both links to sync data for their own “clicks” topic as they would have the same name on the destination cluster. With prefixing, each link can have its own unique prefix that is applied to the topic name as its mirrored. link-1 could have prefix usa_ and link-2 could have prefix eu_. Finally, at the destination cluster there would be two topics, usa_clicks and eu_clicks.

If the link is configured with a prefix, when a mirror topic is created (for example, with confluent kafka mirror create) then the mirror topic name must begin with the prefix (otherwise, the operation will fail). If auto-create mirror topics is used, the topics created on the destination will automatically be named with the prefix.

The prefix can optionally be applied to the consumer groups that are created on the destination cluster because of consumer group offset syncing. When offsets are synced, consumer groups are created on the destination; with this feature it’s possible to prefix the consumer group name on the destination. This enables consumer group offsets to be synced even when two (or more) consumer groups from two (or more) different source clusters have the same name. For example, if link-1 had consumer group g1 and link-2 had consumer group g1, then prefixing would result in two consumer groups at the destination: usa_g1 and eu_g1. By default, consumer group names are not prefixed with the prefix; consumer.group.prefix.enable must be set to true in the cluster link config to enable this.

Here’s an example configuration file for Confluent Enterprise, containing just elements relevant to prefixing:

bootstrap.servers=localhost:9092
cluster.link.prefix=usa_
consumer.offset.sync.enable=true
auto.create.mirror.topics.enable=true
auto.create.mirror.topics.filters={"topicFilters":[{"name": "*","patternType": "LITERAL","filterType": "INCLUDE"}]}
consumer.group.prefix.enable=false
acl.sync.enable=false

Here, a prefix of usa_ has been configured and consumer.group.prefix.enable has been set to false (which is the default, but shown here for context). All mirror topic names on the destination will start with the prefix; consumer group names will remain the same as they are on the source. Also note, acl.sync.enable is set to false which is required because auto.create.mirror.topics.enable is set to true and prefixing is enabled; see limitations below.

On Confluent Cloud, these configurations are specified on the command line or the Confluent Cloud console.

Limitations on Prefixing

The prefix cannot be changed after the cluster link is created.

ACL syncing and prefixing cannot be enabled at the same time. Note, ACLs can always be synced on a separate link; just create a new link and configure it to sync ACLs.

Prefixing cannot be combined with chaining and auto-create mirror topics at the same time. When auto-mirroring and prefixing is configured, a link cannot mirror a topic that is itself a mirror topic at the source cluster. For example, consider the same links above, link-1 and link-2. If a new link-3 was created, auto-mirroring would not be able to mirror data from usa_clicks or eu_clicks or any mirror topic on the destination (even if it didn’t have a prefix) because they are mirror topics. This is done as a safeguard to prevent auto-mirroring from creating an infinite number of topics due to cyclical cluster link connections.

Tip

Prefixed chained mirror topics can still be created by hand, for example via confluent kafka mirror create.

Mirroring Lag

The mirror process is asynchronous in operation. Therefore, there will often be some mirroring lag between the source topic and the mirror topic. The most recent messages on the source topic may not yet have been mirrored to the mirror topic, so the mirror topic may often be slightly behind the source topic.

The same is true for syncing the topic configuration, the consumer group offsets, and the ACLs. All of these processes are asynchronous, so the changes will happen first on the source topic, and then on the mirror topic shortly after.

Syncing consumer group offsets

You can configure your cluster link to sync consumer group offsets. You must pass in a JSON file with a pattern that is matched against consumer group names in order to select which consumer groups to mirror. If these two properties are set, the cluster link syncs the consumer group offsets of any matching consumer groups for all mirror topics that the link mirrors. So, a mirror topic can have consumer group offsets that are synced from the source topic.

Note

If consumer group offset syncing is enabled for a given consumer group name, you should not consume from the mirror topic on the destination cluster with that consumer group name, regardless of whether a consumer group of that name is consuming from the source topic. There is no guaranteed behavior when consuming from a mirror topic with a consumer group name that matches the cluster link’s enabled consumer group name filters.

Why consumer offsets may be clamped in the event of failover, promote, or consumer group migration

When either failover or promote is called on a mirror topic or when a consumer group moves to the destination cluster and consumes its first message from a mirror topic, if consumer offset sync is enabled on the cluster link, then the consumer offsets for that topic may be “clamped”. That is, the consumer offsets for the topic that the cluster link synced will not be allowed to be larger than the last offsets on the mirror topic (the “log end offset”). If any of these consumer offsets are larger / further than the last offsets (Log End Offset), then those consumer offsets will be reset to the Log End Offset. (The use of failover and promote are covered in the next section on Converting a Mirror Topic to a Normal Topic.)

To illustrate with an example: suppose a source topic with one partition had messages up to offset 100 and is being mirrored over a cluster link. However, there was mirroring lag on the cluster link, and only the messages up to offset 90 were mirrored when a disaster hit the source topic. At this point, you call failover on the mirror topic. Consumer group A was at offset 80 on the source cluster, so it will remain at offset 80 on the destination cluster, since that offset was mirrored to the mirror topic. But consumer group B was at offset 95, which was not mirrored to the mirror topic. If consumer group B started consuming at offset 95 on the mirror topic, then it would miss any messages at offsets 90-94 that were produced to the topic. To avoid that problem, the cluster link “clamps” consumer group B’s offsets down to 90, which is the highest offset on the mirror topic.

To learn more, see consumer.offset.group.filters in the Confluent Cloud documentation under Configuring Cluster Link Behavior and in the Confluent Platform documentation under configuration options for cluster links.

Converting a Mirror Topic to a Normal Topic

If you want to convert a mirror topic into a normal topic that you can produce into, you can call the failover or the promote command on the mirror topic.

In Confluent Platform, you call kafka-mirrors --promote or kafka-mirrors --failover.

In Confluent Cloud, you call confluent kafka mirror promote <topic-name> or confluent kafka mirror failover <topic-name>.

Both commands occur on the destination cluster (the mirror topic’s cluster) and require you to pass in the cluster link’s name.

The promote option is often used for migrations. It checks that there is no mirroring lag, config sync lag, or consumer offset lag between the source topic and the mirror topic. Then, it converts the mirror topic into a full topic, with the assurance that this topic was exactly the same as its source topic.

Note

The destination cluster’s brokers must be able to reach the source cluster’s brokers in order to make this check, so your source cluster must be online.

The failover option is often used when a disaster has hit the source cluster (for example, a cloud region outage) and you want to shift operations from the source topic to the mirror topic. This command will succeed regardless of the mirroring lag or the source cluster’s reachability.

Both the promote and failover commands are irreversible. There is no way to change this topic back into a mirror topic. If you want a mirror topic of the same name as the one you promoted or failed over on this cluster, you must delete the converted topic, and create a new mirror topic of the same name.

There is no way to change a mirror topic to use a different cluster link or make changes to the link itself, other than to recreate the mirror topic on a different link.

Important

  • You can run mirror describe (confluent kafka mirror describe <mirror-topic-name> --link <link>) on a promoted or failed over mirror topic, if you don’t delete the cluster link. If you delete the cluster link, you will lose the history and, therefore, mirror describe will not find data on promoted or failed over topics.
  • You cannot delete a cluster link that still has mirror topics on it (the delete operation will fail).
  • If you are using Confluent for Kubernetes (CFK), and you delete your cluster link resource, any mirror topics still attached to that cluster link will be forcibly converted to regular topics by use of the failover API. To learn more, see Modify a mirror topic in Cluster Linking using Confluent for Kubernetes.
../../_images/cluster-link-migrate.png

Example of Topic Migration

../../_images/cluster-link-failover.png

Example of Failing Over a Topic

Mirror Topic States and Statuses

When you describe a mirror topic, it will return one of these states:

ACTIVE
The mirror is running normally, and messages are being mirrored from the source topic to the destination topic.
PAUSED
  • A user has paused mirroring for this mirror topic.
  • To reach this state, a user must either pause this specific topic, or pause its cluster link.
PENDING_STOPPED
  • A user has stopped this mirror topic with the promote command, and this topic will soon be in the STOPPED state.
  • To force the mirror topic to immediately go from the PENDING_STOPPED state to the STOPPED state, call the failover command on it. Doing this cancels any synchronization that was happening between the source cluster and the destination cluster, and eliminates any guarantees that the promote command gives.
STOPPED
  • Mirroring has permanently stopped for this topic. It will no longer receive messages from its source topic. The topic is now writable and can receive messages produced directly to it.
  • To get into this state, a user must call either promote or failover on this mirror topic.
  • Even though a STOPPED topic is no longer a mirror topic, it will still be listed in output for the commands confluent kafka mirror list and confluent kafka mirror describe <destination-topic-name> --link <link> for as long as the cluster link exists. This is useful because the topic will return the last offset it fetched from its source topic (Last Source Fetch Offset) for each partition, and the time at which it was stopped (Status Time).
SOURCE_UNAVAILABLE
  • The mirror topic is unable to reach the source topic, and is not mirroring messages from the source topic. This could happen if the source cluster is experiencing an outage or if the network between the destination cluster and the source cluster is unstable.
  • Mirroring will resume once the issue is resolved and the destination cluster can reach the source cluster.

Note

Using a Confluent Platform 7.0.x source cluster with a source-initiated link to a KRaft destination cluster will generate a SOURCE_UNAVAILABLE error. Cluster Linking between a source cluster running Confluent Platform 7.0.x or earlier (non-KRaft) and a destination cluster running in KRaft mode is not supported. To solve for this, upgrade the source cluster to Confluent Platform 7.1.0 or later.

LINK_FAILED
  • An error has broken the mirror topic’s cluster link, and no data is being mirrored. A user needs to manually re-configure the link.
FAILED
  • The mirror topic has permanently failed. It will no longer mirror data. This can happen if the cluster link ACLs are removed from the source cluster, or if the source topic is deleted. In both cases, the failed status takes effect only after cluster.link.retry.timeout.ms is reached (by default, the system retries the link for 5 minutes).
  • You can stop this mirror with the failover command, and it will become a regular topic.
  • If you want to restore mirroring for this topic, you must delete the legacy mirror topic and create a new mirror topic with the same name.

Mirror Topic Deletion

You may safely delete a mirror topic. Deleting a mirror topic permanently stops data mirroring to that topic. If you create a new normal topic of the same name on the same cluster, data will not be mirrored to it.

To delete a mirror topic, use the same command you would use to delete a normal topic:

confluent kafka topic delete <topic-name>

To learn more, see Deleting topics that were auto-created.

Note

You cannot delete a cluster link that is attached to any mirror topics. You must first delete, failover, or promote all of the mirror topics, and then you can delete the cluster link.

Source Topic Deletion

Caution

Do not delete a source topic that is being mirrored by a mirror topic. Doing so can lead to unpredictable truncation and data loss on the mirror topic. You should always stop mirroring to all associated mirror topics before deleting a source topic.

While it may be possible to delete a source topic that is being mirrored by a mirror topic and a cluster link, doing so is not recommended. In particular, unpredictable behavior can occur if a source topic is deleted, and a topic by the same name is then created within a few minutes time. This scenario can cause permanent data loss on any mirror topics that are still mirroring from that source topic, and can also cause performance issues on the source cluster or destination cluster.

Before deleting a source topic, you should stop any mirroring to associated mirror topics. You can stop mirroring on a mirror topic in one of these ways:

  • Delete the mirror topic.
  • Call promote or failover so the mirror topic enters the STOPPED state.
  • Revoke the security permissions for the cluster link to read the mirror topic. You can do this in one of three ways: (1) delete the cluster link’s ALLOW ACL for the source topic, (2) create a DENY ACL for the source topic, or (3) delete the cluster link’s API key.

For further discussion about Kafka’s limitations with topic deletion and how topic IDs will help, see KIP-516.

How Schemas work with Mirror Topics

Cluster Linking preserves the schema ID stored in each message. Therefore, to consume from a mirror topic that is using schemas, the consumer clients must use a Schema Registry context with the same schema IDs as on the Schema Registry context used by the producers to the source topic. Consequently, consuming from a mirror topic that uses schemas should be done either by:

  • (Option 1) using the same Schema Registry as the producers used
  • (Option 2) using a Schema Registry context that was synced through Schema Linking from the Schema Registry that the producers used

Caution

When using Schema Linking: To use a mirror topic that has a schema with Confluent Cloud Connect, ksqlDB, broker-side schema validation, or the topic viewer, make sure that Schema Linking puts the schema in the default context of the Confluent Cloud Schema Registry. These fully-managed Confluent Cloud features require schemas to be in the default context of the Confluent Cloud Schema Registry in their Environment.

../../_images/mirror-topics-and-schemas.png

Mirror Topics and Schemas

Advanced Mirror Topic Architectures

Chaining

A mirror topic can be a source topic itself. A mirror topic can be mirrored by different cluster links, allowing you to “chain” cluster links and mirror topics together.

For example, Topic A (source topic) on Cluster 1 —cluster link—> Topic A (mirror topic and source topic) on Cluster 2 —cluster link—> Topic A (mirror topic) on Cluster 3

Tip

You can safely create these chained topics and cluster links without creating a circular dependency between mirror topics and cluster links. You can create mirror topics without the fear of creating an infinite loop.

../../_images/cluster-link-chain.png

Chaining Example

Fanning Out

A source topic can be mirrored to multiple mirror topics. These mirror topics must exist on multiple different clusters.

For example, Topic A on Cluster 1 —cluster link—> Topic A on Cluster 4, and Topic A on Cluster 1 —cluster link—> Topic A on Cluster 5

Tip

If you plan to use failover or promote on a cluster link (for example, for Disaster Recovery or Migration), then chained or fanned-out mirror topics will not automatically retain their shape. For example, if you fan out A –> B and A –> C, if A has an outage and you call failover on B, there is no way to automatically mirror B –> C. You will need to reconstruct the appropriate mirroring relationship for your use case using brand new topics.

../../_images/cluster-link-fan-out.png

Fan-Out Example

Configurations

The following sections provide a quick reference of which Cluster Linking configurations are synced from the source to the mirror topic, overrides, and concepts related to syncing. For a full configuration reference, see Configuration Options for Cluster Linking.

Synced Mirror Topic Configurations for Confluent Platform

These configurations are always synced from the source topic to the mirror topic. Mirror topics will always have the same value as their source topic, in order to ensure the properties of mirror topics are met.

  • number of partitions
  • max.message.bytes
  • cleanup.policy
  • retention.bytes
  • retention.ms
  • delete.retention.ms
  • min.compaction.lag.ms
  • max.compaction.lag.ms
  • message.timestamp.type
  • message.timestamp.difference.max.ms
  • compression.type

Overridable Mirror Topic Configurations

These configurations are configurable in Confluent Platform, meaning you can override the source configurations for these on the mirror topic.

  • segment.jitter.ms
  • segment.index.bytes
  • flush.messages
  • flush.ms
  • index.interval.bytes
  • min.cleanable.dirty.ratio
  • file.delete.delay.ms
  • preallocatemessage.format.version
  • confluent.segment.speculative.prefetch.enable

Mirror Topic Configurations Not Synced for Confluent Platform

Any configuration that is not in the list above will not be synced to a mirror topic in Confluent Platform. Therefore, the mirror topic’s configuration could be different from the source topic’s configuration. If you don’t override the mirror topic’s configuration, then it will inherit its cluster’s default.

A few important examples of configurations that are not synced to mirror topics in Confluent Platform:

  • min.insync.replicas

  • confluent.placement.constraints

  • confluent.tier.enable

  • confluent.key.schema.validation

  • confluent.value.schema.validation

  • replication.factor

    Tip

    No replication factors are synced to mirror topics. Mirror topics use the default.replication.factor configured on the brokers. If not explicitly set, this defaults to a replication factor of 1, and consequently the mirror topics will pick up a replication factor of 1. For a description of this option, see default.replication.factor in Kafka Broker Configurations.

Hybrid Cloud Configuration Syncs

Confluent Platform and Confluent Cloud have different policies for which mirror topic configurations are synced. If you cluster link between Confluent Platform and Confluent Cloud, then the destination cluster’s policy is enforced.

For example, if you cluster link from a Confluent Platform source cluster to a Confluent Cloud destination cluster, then the value of compression.type will not be synced. But if you cluster link from a Confluent Cloud source cluster to a Confluent Platform destination cluster, then compression.type will be synced.