Cluster Linking Overview
Looking for Confluent Platform Cluster Linking docs? This page describes
Cluster Linking on Confluent Cloud. If you are looking for the Confluent Platform documentation,
check out Cluster Linking on Confluent Platform.
This feature is available as a preview feature. A preview feature is a component of Confluent Cloud that is being
introduced to gain early feedback from developers. These features can be used for evaluation and
non-production testing purposes or to provide feedback to Confluent. Comments, questions, and suggestions
related to preview features are encouraged and can be submitted to email@example.com.
What is Cluster Linking?
Cluster Linking on Confluent Cloud is a fully-managed service for moving data
from one Confluent cluster to another. Programmatically, it creates perfect
copies of your topics and keeps data in sync across clusters. Cluster Linking is
a powerful geo-replication technology for:
- Multi-cloud and global architectures powered by real-time data in motion
- Data sharing between different teams and lines of business
- Hight Availability (HA)/Disaster Recovery (DR) during a regional cloud provider outage
- Data and workload migration from a Apache Kafka® cluster to Confluent Cloud
Cluster Linking is fully-managed in Confluent Cloud, so you don’t need to manage
or tune data flows. Its usage-based pricing puts multi-cloud and multi-region
costs into your control. Cluster Linking reduces operational burden and cloud
egress fees, while improving the performance and reliability of your cloud data
How it Works
Cluster Linking allows one Confluent cluster to mirror data directly from
another. You can establish a cluster link between a source cluster and a
destination cluster in a different region, cloud, line of business, or
organization. You choose which topics to replicate from the source cluster to
the destination. You can even mirror consumer offsets and ACLs, making it
straightforward to move Kafka consumers from one cluster to another.
In one command or API call, you can create a cluster link from one cluster to
another. A cluster link acts as a persistent bridge between the two clusters.
ccloud kafka link create tokyo-sydney
To mirror data across the cluster link, you create mirror topics on your destination cluster.
ccloud kafka mirror create clickstream.tokyo
Mirror topics are a special kind of topic: they are read-only copies of their
source topic. Any messages produced to the source topic are mirrored to the
mirror topic “byte-for-byte,” meaning that the same messages go to the same
partition and same offset on the mirror topic. Mirror topics can be consumed
from just the same as any other topic.
Cluster links and mirror topics are the building blocks you can use to create
scalable, consistent architectures across regions, clouds, teams, and organizations.
Confluent provides multi-cloud, multi-region, and hybrid capabilities in Confluent Cloud.
Some of these are demo’ed in the Tutorials.
- Global Data Sharing - Share data for selected topics across different regions, clouds, environments, or teams.
- Data Migration - Migrate data and workloads from one cluster to another.
- Disaster Recovery and High Availability - Create a disaster recovery cluster, and fail over to it during an outage.
Cluster Linking mirroring throughput (the bandwidth used to read data or write data to your cluster) is counted against your Dedicated cluster CKUs and limits.
A cluster link sends data from a “source cluster” to a “destination cluster”.
How to Check the Cluster Type
To check a Confluent Cloud cluster’s type and endpoint type:
Log on to Confluent Cloud.
Select an environment.
Select a cluster.
The cluster type is shown on the summary card for the cluster.
Alternatively, click into the cluster, and select Cluster settings from the left menu.
The cluster type is shown on the summary card for “Cluster type”.
From within Cluster settings for a dedicated cluster, click the Networking tab to view the endpoint type.
Only Dedicated clusters have the Networking tab; Basic and Standard
clusters always have Internet networking. Networking is defined when you first create the Dedicated cluster.
Cluster Linking is not supported for Confluent Cloud clusters that have the Transit
Gateway, VPC Peering, Privatelink or VNet Peering networking types. If you wish
to use Cluster Linking with a privately networked Confluent Cloud cluster,
contact your Confluent account team or email firstname.lastname@example.org to find out more.
Cluster Linking does not currently support aggregating data from more than
five different source clusters into one destination cluster.
During the Preview, there is no extra charge to use Cluster Linking . Mirroring
throughput (reading from and writing to a cluster using a cluster link) costs
the same as consuming from and producing data to a Confluent Cloud cluster
Just getting started with Cluster Linking? Here are a few suggestions for next steps.
To get started, try one or more tutorial, each of which maps to a use case.
Read-only, mirror topics that reflect the data in original (source) topics are the building blocks of Cluster Linking.
For a deep dive on this specialized type of topic and how it works, see Mirror Topics.
Commands and Prerequisites
The destination cluster can use the
ccloud kafka link command to create a link from the source cluster.
The following prerequisite steps are needed to run the tutorials during the Preview.
To try out Cluster Linking on Confluent Cloud:
Log on to Confluent Cloud.
Update your Confluent Cloud CLI with
ccloud update to be sure you have an up-to-date version
ccloud kafka link command.
ccloud kafka link command has the following subcommands or flags.
|Create a new cluster link.
|Delete a previously created cluster link.
|Describes an existing cluster link.
|Lists existing cluster links.
|Updates a property for an existing cluster link.
Type the following in the same command shell you are using for Confluent Cloud.
Follow the tutorials to try out Cluster Linking. The commands are demo’ed in the tutorials.