Cluster Linking

Looking for Confluent Cloud Cluster Linking docs? You are currently viewing Confluent Platform documentation. If you are looking for Confluent Cloud docs, check out Cluster Linking on Confluent Cloud.

What is Cluster Linking?

Cluster Linking enables you to directly connect clusters and mirror topics from one cluster to another. Cluster Linking makes it easy to build multi-datacenter, multi-region, and hybrid cloud deployments. It is secure, performant, tolerant of network latency, and built into Confluent Server and Confluent Cloud.

Unlike Replicator and MirrorMaker 2, Cluster Linking does not require running Connect to move messages from one cluster to another, and it creates identical “mirror topics” with globally consistent offsets. We call this “byte-for-byte” replication. Messages on the source topics are mirrored precisely on the destination cluster, at the same partitions and offsets.

Capabilities and Comparisons

Cluster Linking replicates topics from one Kafka or Confluent cluster to another, providing the following capabilities:

  • Global Replication: Unify data and applications from regions and continents around the world.
  • Hybrid cloud: Create a secure, scalable, and seamless bridge-to-cloud by linking an on-premise Confluent Platform cluster in a private cloud to a Confluent Cloud cluster in a public cloud.
  • HA/DR: Build a multi-region high availability and disaster recovery (“HA/DR”) strategy that achieves low recovery times (RTOs) and minimal data loss (RPOs) by replicating topic data and metadata to another cluster.
  • Cluster migration: Migrate from an older cluster to one in a newer environment, region, or cloud.
  • Aggregation: Combine data from many smaller clusters into one aggregate cluster.
  • Data sharing: Exchange data between different teams, lines-of-business, and organizations.

Compared to other Kafka replication options, Cluster Linking offers these advantages:

  • Built into Confluent Server and Confluent Cloud, so it does not depend on additional components, connectors, virtual machines, or custom processes.
  • Creates exact mirrors of topics, including offsets, to enable migration, failover, and rationalizing about your system without offset translation or custom tooling.
  • Can be dynamically updated via REST APIs, CLIs, and Kubernetes CRDs.
  • For compressed messages, byte-to-byte replication achieves faster throughput by avoiding decompression-and-recompression.

What’s Supported

Cluster Linking is included as a part of Confluent Server. There are no additional or other licensing costs for Cluster Linking on Confluent Platform outside of the cost of the Confluent Enterprise Enterprise (Subscription) License. Following are the requirements for and supported features of Cluster Linking.

  • Requires Confluent Server destination cluster of Confluent Platform 7.x.x on the destination cluster
  • Works with all clients
  • Requires an inter-broker protocol (IBP) of 2.4 or higher on the source cluster, and an IBP of 2.7 or higher on the destination cluster. More specifically, if the destination cluster is versions 7.0.0-7.0.3 or 7.1.0-7.1.1 and the IBP of source cluster is 2.7 or lower, IBP of destination must be 2.7. If IBP of source cluster is 2.8 or higher, IBP of destination may be 2.7 or higher. It is not supported for clusters using Confluent Platform 7.0.0-7.0.3 or 7.1.0-7.1.1 to have a mismatch between the source and destination IBP versions specified. For a guide to upgrading, see Steps for upgrading to 7.0.x.
  • Requires password.encoder.secret to be set on the Confluent Platform destination cluster’s brokers. To learn how to set this property dynamically, see Updating Password Configurations Dynamically.
  • Built-in custom resource in Confluent for Kubernetes
  • Compatible with Ansible. To learn more, see Using Cluster Linking with Ansible.
  • Provides support for authentication and authorization, as described in Cluster Linking Security.
  • The source cluster can be Kafka or Confluent Server or Confluent Cloud; the destination cluster must be Confluent Server, which is bundled with Confluent Enterprise.
  • In addition to self-managed deployments on Confluent Platform, Cluster Linking is also available as a managed service on Confluent Cloud and in Hybrid Cloud.
Source Destination
Confluent Platform 5.4 or later Confluent Platform 7.0.0 or later
Confluent Cloud Confluent Platform 7.0.0 or later
Kafka 2.4 or later Confluent Platform 7.0.0 or later
Confluent Platform 5.4 or later Confluent Cloud [1]
Confluent Cloud Confluent Cloud [1]
Kafka 2.4 or later Confluent Cloud [1]

Footnotes

[1](1, 2, 3) See Cluster Linking on Confluent Cloud for supported Confluent Cloud source and destination combinations.

Upgrade Notes

  • When upgrading from Confluent Platform 6.2.0 to 7.0.0 or later version, make sure that acl.sync.enabled is not set to true in $CONFLUENT_HOME/server.properties. (The default is false, so if this property is not specified, you can assume it is set to false.) If acl.sync.enabled is set to true during upgrade, existing cluster links will be marked as “failed” (state is LINK_FAILED).

Use Cases and Architectures

The following use cases can be achieved by the configurations and architectures shown.

Hybrid Cloud

Use Case: Easily create a persistent and seamless bridge from on-premise environments to cloud environments. A cluster link between a Confluent Platform cluster in your datacenter and a Confluent Cloud cluster in a public cloud acts as a single secure, scalable hybrid data bridge that can be used by hundreds of topics, applications, and data systems. Cluster Linking can tolerate the high latency and unpredictable networking availability that you might have between on-premise infrastructure and the cloud, and recovers from reconnections automatically. Cluster Linking can replicate data bidirectionally between your datacenter and the cloud without any firewall holes or special IP filters because your datacenter always makes an outbound connection. Cluster Linking creates a byte-for-byte, globally consistent copy of your data that preserves offsets, making it easy to migrate on-premise applications to the cloud. Cluster Linking built into Confluent Platform and does not require any additional components to manage.

../../_images/clusterlinking-usecase-hybrid.png

Disaster Recovery

Use Case: Create a Disaster Recovery (“DR”) cluster that is available to failover should your primary cluster experience an outage or disaster. Cluster Linking keeps your DR cluster in sync with data, metadata, topic structure, topic configurations, and consumer offsets so that you can achieve low recovery point objectives (“RPOs”) and recovery time objectives (“RTOs”), often measured in minutes. Cluster Linking for DR does not require an expensive network, complicated management, or extra software components. And because Cluster Linking preserves offsets and syncs consumer offsets, consumer applications of all languages can failover and pickup near the point where they left off, achieving low downtime without custom code or interceptors.

../../_images/clusterlinking-usecase-dr.png

Global Replication

Use Case: Stream data between the continents and regions where your business operates. Unify data from every region to create a global real-time event mesh. Aggregate data from different regions to drive the real-time applications and analytics that power your business. By making geo-local reads of real-time data possible, this can act like a content delivery network (CDN) for your Kafka events throughout the public cloud, private cloud, and at the edge.

../../_images/clusterlinking-usecase-global.png

Data Sharing

Use Case: Share data between different teams, lines of business, or organizations in a pattern that provides high isolation between teams and efficient operational management. Cluster Linking keeps an in-sync mirror copy of relevant data on the consuming team’s cluster. This isolation empowers the consuming team to scale up hundreds of consumer applications, stream processing apps, and data sinks without impacting the producing team’s cluster: for the producing team, it’s the same load as one additional consumer. The producing team simply issues a security credential with access to the topics that the consuming team is allowed to read. Then the consuming team can create a cluster link, which they control, monitor, and manage.

../../_images/clusterlinking-usecase-datasharing.png

Cluster Migration

Use Case: Seamlessly move from an on-premises Kafka or Confluent Platform cluster to a Confluent Cloud cluster, or from older infrastructure to new infrastructure, with low downtime and no data loss. Cluster Linking’s native offset preservation and consumer offset syncing allows every consumer application to switch from the old cluster to the new one when it’s ready. Topics can be migrated over one by one, or in a batch. Cluster Linking handles topic creation, configuration, and syncing.

../../_images/clusterlinking-usecase-migration.png

Known Limitations and Best Practices

  • In Confluent Platform 7.0.0, source initiated cluster links (recommended for hybrid cloud) cannot be created with the REST API; they must be created with the kafka-cluster-links CLI. Regular, destination initiated cluster links can be created with either the REST API or kafka-cluster-links CLI.
  • In Confluent Platform 7.0.x, REST API calls to list and get source-initiated cluster links will have their destination cluster IDs returned under the parameter source_cluster_id.
  • All SSL key stores, trust stores and Kerberos keytab files must be stored at the same location on each broker in a given cluster. If not, cluster links may fail. Alternatively, you can configure a PEM certificate in-line on the cluster link configuration.
  • Cluster link configurations stored in files (SSL key stores, trust stores, Kerberos keytab files) should not be stored in /tmp because /tmp files may get deleted, leaving links and mirrors in a bad state on some brokers.
  • Confluent Control Center will only display mirror topics correctly if the Confluent Platform cluster and Control Center are connected to a REST Proxy API v3. If not connected to the v3 Confluent REST API, Control Center will display mirror topics as regular topics, which can lead to showing features that are not actually available on mirror topics; for example, producing messages or editing configurations. To learn how to configure these clusters for the v3 REST API, see Required Configurations for Control Center.
  • Prerequisites are provided per tutorial or use case because these differ depending on the context. Tutorials are provided on topic data sharing and Hybrid Cloud and Bridge-to-Cloud. Additional requirements for secure setups are provided in Cluster Linking Security.
  • Cluster Linking is not integrated with Kafka transactions, including “exactly once” semantics. Using Cluster Linking to mirror topics that contain transactions or exactly once semantics is not supported and not recommended.
  • Cluster Linking currently does not work with KRaft. Cluster Linking requires ZooKeeper.
  • ACL migration (ACL sync), previously available in Confluent Platform 6.0.0 through 6.2.x, was removed due to a security vulnerability. If you are using ACL migration in your deployments, you should disable it. To learn more, see Authorization (ACLs).