Frequently Asked Questions (FAQs)¶
Looking for Confluent Platform Cluster Linking docs? You are currently viewing Confluent Cloud documentation. If you are looking for Confluent Platform docs, check out Cluster Linking on Confluent Platform.
FAQ Quick List¶
- How do I know the data was successfully copied to the destination and can be safely deleted from the source topic?
- How can I throttle a Confluent Platform to Confluent Cloud cluster link to make the best use of my bandwidth?
- How is the consumer offset sync accomplished?
- How do I create a cluster link?
- Which clusters can create cluster links?
- How do I create a mirror topic?
- Can I prevent certain topics from being mirrored by a cluster link?
- Can I use Cluster Linking without the traffic going over the public internet?
- I need RPO==0 (guarantee of no data loss after a failover) in Confluent Cloud. What can I do?
- If I want to join two topics from different clusters in ksqlDB, how can Cluster Linking help me?
- Does Cluster Linking work with mTLS?
- How does Schema Registry multi-region disaster recovery (DR) work in Confluent Cloud?
- How can I automatically failover Kafka clients?
- How does Cluster Linking optimize network bandwidth and performance in Confluent Cloud?
- How do I perform a failover on a cluster link used primarily for data sharing?
How do I know the data was successfully copied to the destination and can be safely deleted from the source topic?¶
Each mirror topic has a metric called “Max Lag” which is the maximum number of messages any of its partitions are lagging by. You can monitor this through the Metrics API, Confluent Cloud Console, or Confluent Control Center.
If you need partition-level guarantees, you can use the REST API or the CLI to describe a mirror topic and see the exact offset that each partition has replicated up to. It will be called “Last Source Fetch Offset.” Anything before that offset in that partition has been replicated.
How is the consumer offset sync accomplished?¶
Consumer offset sync does not mirror the internal topic
_consumer_offsets, but rather uses
code for the controller nodes to sync the offsets on a frequency that you can configure.
See “Syncing consumer group offsets” in
mirror topics on Confluent Cloud
and mirror topics on Confluent Platform.
How do I create a mirror topic?¶
You can create a specific mirror topic using the Confluent Cloud Console, the Confluent Cloud REST API, the Confluent CLI, the Confluent Platform AdminClient API, or Confluent for Kubernetes. Alternatively, you can configure your cluster link to automatically create mirror topics that match certain prefixes. To learn more, see Mirror Topic Creation.
I need RPO==0 (guarantee of no data loss after a failover) in Confluent Cloud. What can I do?¶
Achieving RPO==0 with Confluent Cloud is possible with correctly architected producer clients. Cluster Linking does not have an RPO==0 (also known as “synchronous”) option out-of-the-box. The way that users have achieved RPO==0 with async tech like Cluster Linking is for producers to have a durable, multi-region cache of their most recent data. This cache might store the past 15 minutes of data, for example. The producers also have a failover mechanism that causes them to replay all the cached data; or this data can be replayed by the platform operator at the start of a failover. This will result in duplicate events in the underlying topics, which your consumers should be able to tolerate anyways, because that is a requirement for any disaster recovery (DR) scenario where the consumers failover.
How does Schema Registry multi-region disaster recovery (DR) work in Confluent Cloud?¶
Confluent Cloud Schema Registry is a three node cluster spanning three different zones within a region (multi-zone), meaning it can tolerate two node failures in that region. For multi-region DR, you can use Schema Linking to replicate and keep in sync schemas across two Schema Registry clusters in different regions.
How can I automatically failover Kafka clients?¶
On Confluent Cloud: A detailed solution for how to failover clients as quickly as possible will be published as a whitepaper. Please check back on these FAQs in a few weeks.
On Confluent Platform, customers have more control than those on Confluent Cloud. This gives you an additional option:
- Use a Domain Name System (DNS) service (such as Amazon Route 53) to direct Kafka clients to the “active” cluster. In this scenario, the bootstrap server points to a URL, and you update the DNS to have that URL resolve to the “active” cluster. Kafka clients do not need to change their bootstrap server on a failover.
- Use the same authentication information on both clusters; so Kafka clients do not need to change their security configuration on a failover.
So, to failover clients, you simply change the DNS. When the clients start up, they will bootstrap to the “active” cluster.
The only thing to keep in mind here is that bootstrapping only happens when a client starts up. (This is an advantage of Multi-Region Clusters, since it avoids the re-bootstrapping process.) If you have any clients that are still running and survived the outage, they will need to have a way to restart in order to failover. This is less of a concern in Confluent Cloud, for two reasons:
- Often a restart can be called by the Operator, if all clients are running on K8’s (Kubernetes) clusters.
- On Confluent Cloud, the outage is a cloud service provider (CSP) outage in a given region. Let’s say your clients are all in the same region as the cluster they point to. For example, for an Amazon AWS cluster, suppose all clients are also on AWS and in the same region as the cluster. If there is an AWS outage in that region, no clients will survive the outage; they will all need to be restarted anyways, by definition.