Cluster Linking Overview

Looking for Confluent Platform Cluster Linking docs? This page describes Cluster Linking on Confluent Cloud. If you are looking for the Confluent Platform documentation, check out Cluster Linking on Confluent Platform.

Important

This feature is available as a preview feature. A preview feature is a component of Confluent Cloud that is being introduced to gain early feedback from developers. These features can be used for evaluation and non-production testing purposes or to provide feedback to Confluent. Comments, questions, and suggestions related to preview features are encouraged and can be submitted to clusterlinking@confluent.io.

What is Cluster Linking?

Cluster Linking on Confluent Cloud is a fully-managed service for moving data from one Confluent cluster to another. Programmatically, it creates perfect copies of your topics and keeps data in sync across clusters. Cluster Linking is a powerful geo-replication technology for:

  • Multi-cloud and global architectures powered by real-time data in motion
  • Data sharing between different teams and lines of business
  • Hight Availability (HA)/Disaster Recovery (DR) during a regional cloud provider outage
  • Data and workload migration from a Apache Kafka® cluster to Confluent Cloud

Cluster Linking is fully-managed in Confluent Cloud, so you don’t need to manage or tune data flows. Its usage-based pricing puts multi-cloud and multi-region costs into your control. Cluster Linking reduces operational burden and cloud egress fees, while improving the performance and reliability of your cloud data pipelines.

How it Works

Cluster Linking allows one Confluent cluster to mirror data directly from another. You can establish a cluster link between a source cluster and a destination cluster in a different region, cloud, line of business, or organization. You choose which topics to replicate from the source cluster to the destination. You can even mirror consumer offsets and ACLs, making it straightforward to move Kafka consumers from one cluster to another.

../../_images/cloud-cluster-linking-overview.png

In one command or API call, you can create a cluster link from one cluster to another. A cluster link acts as a persistent bridge between the two clusters.

ccloud kafka link create tokyo-sydney
  --source-bootstrap-server pkc-867530.ap-northeast-1.aws.confluent.cloud:9092
  --source-cluster-id lkc-42492
  --api-key AP1K3Y
  --api-secret ********

To mirror data across the cluster link, you create mirror topics on your destination cluster.

ccloud kafka mirror create clickstream.tokyo
   --link tokyo-sydney
../../_images/cloud-cluster-linking-mirror.png

Mirror topics are a special kind of topic: they are read-only copies of their source topic. Any messages produced to the source topic are mirrored to the mirror topic “byte-for-byte,” meaning that the same messages go to the same partition and same offset on the mirror topic. Mirror topics can be consumed from just the same as any other topic.

../../_images/cloud-cluster-linking-mirror-topics.png

Cluster links and mirror topics are the building blocks you can use to create scalable, consistent architectures across regions, clouds, teams, and organizations.

Use Cases

Confluent provides multi-cloud, multi-region, and hybrid capabilities in Confluent Cloud. Some of these are demo’ed in the Tutorials.

  • Global Data Sharing - Share data for selected topics across different regions, clouds, environments, or teams.
  • Data Migration - Migrate data and workloads from one cluster to another.
  • Disaster Recovery and High Availability - Create a disaster recovery cluster, and fail over to it during an outage.

Cluster Linking mirroring throughput (the bandwidth used to read data or write data to your cluster) is counted against your Dedicated cluster CKUs and limits.

Cluster Types

A cluster link sends data from a “source cluster” to a “destination cluster”.

Source Cluster Options Destination Cluster Options
  • Dedicated Confluent Cloud cluster with Internet networking
  • Apache Kafka® 2.4+ or Confluent Platform 5.4+
  • Dedicated Confluent Cloud cluster with Internet networking capable of reaching source cluster brokers

How to Check the Cluster Type

To check a Confluent Cloud cluster’s type and endpoint type:

  1. Log on to Confluent Cloud.

  2. Select an environment.

  3. Select a cluster.

    The cluster type is shown on the summary card for the cluster.

    ../../_images/cloud-cluster-link-dedicated-cluster.png

    Alternatively, click into the cluster, and select Cluster settings from the left menu. The cluster type is shown on the summary card for “Cluster type”.

    ../../_images/cloud-cluster-link-settings.png

    From within Cluster settings for a dedicated cluster, click the Networking tab to view the endpoint type. Only Dedicated clusters have the Networking tab; Basic and Standard clusters always have Internet networking. Networking is defined when you first create the Dedicated cluster.

    ../../_images/cloud-cluster-link-dedicated-cluster-networking.png

Limitations

Cluster Linking is not supported for Confluent Cloud clusters that have the Transit Gateway, VPC Peering, Privatelink or VNet Peering networking types. If you wish to use Cluster Linking with a privately networked Confluent Cloud cluster, contact your Confluent account team or email clusterlinking@confluent.io to find out more.

Cluster Linking does not currently support aggregating data from more than five different source clusters into one destination cluster.

Pricing

During the Preview, there is no extra charge to use Cluster Linking . Mirroring throughput (reading from and writing to a cluster using a cluster link) costs the same as consuming from and producing data to a Confluent Cloud cluster (KafkaNetworkRead and KafkaNetworkWrite).

First Look

Just getting started with Cluster Linking? Here are a few suggestions for next steps.

Tutorials

To get started, try one or more tutorial, each of which maps to a use case.

Mirror Topics

Read-only, mirror topics that reflect the data in original (source) topics are the building blocks of Cluster Linking. For a deep dive on this specialized type of topic and how it works, see Mirror Topics.

Commands and Prerequisites

The destination cluster can use the ccloud kafka link command to create a link from the source cluster. The following prerequisite steps are needed to run the tutorials during the Preview.

To try out Cluster Linking on Confluent Cloud:

  1. Log on to Confluent Cloud.

  2. Update your Confluent Cloud CLI with ccloud update to be sure you have an up-to-date version of the ccloud kafka link command.

    The ccloud kafka link command has the following subcommands or flags.

    Command Description
    create Create a new cluster link.
    delete Delete a previously created cluster link.
    describe Describes an existing cluster link.
    list Lists existing cluster links.
    update Updates a property for an existing cluster link.
  3. Type the following in the same command shell you are using for Confluent Cloud.

    export XX_CCLOUD_USE_KAFKA_REST=true
    
  4. Follow the tutorials to try out Cluster Linking. The commands are demo’ed in the tutorials.

Suggested Resources