Manage Cluster Linking for Confluent Platform Using Confluent for Kubernetes

The Cluster Linking feature allows you to directly connect Kafka clusters together and mirror topics from one cluster (the source) to another (the destination). You can use the feature in multi-datacenter, multi-cluster, and hybrid cloud deployments to seamlessly move data between Kafka clusters.

Confluent for Kubernetes (CFK) provides a declarative API, the ClusterLink custom resource definition (CRD), for defining, updating, and managing cluster links.

For use case scenarios for configuring cluster links in CFK, refer to Cluster Linking Use Cases in the CFK example GitHub repo.

The following types of cluster links are supported in CFK. Select a type of cluster links based on where you want to move data to and from, and configure the ClusterLink CR accordingly:

  • Confluent Platform to Confluent Platform cluster links

    To move data from an on-prem Confluent Platform to another on-prem Confluent Platform or from Confluent Cloud to Confluent Cloud, create a cluster link on the destination with connection and security information for the destination to talk to the source. No changes are required on the source cluster.

    The destination cluster initiates the connection to the source cluster.

    You can also use this type of cluster links to migrate data from Apache Kafka® or from other Kafka, such as Amazon MSK, to on-prem Confluent Platform.

    In this document, this type of cluster link is referred to as destination-initiated. Note that this is not a term used in the other cluster linking documentation.

  • Confluent Platform to Confluent Cloud cluster links

    To move data from an on-prem Confluent Platform to Confluent Cloud in a hybrid environment, use the source-initiated type of cluster links.

    For source-initiated cluster links, you configure connection and security information on the source cluster to talk to the destination.

    The source cluster (on-prem Confluent Platform) initiates the connection to the destination cluster (Confluent Cloud).

    For source-initiated cluster links, cluster links need to be configured both on the source and destination clusters.

For information on how to create each type of cluster link, see Create a cluster link.

Requirements and considerations

Before you create a cluster link, review the following requirements and considerations:

  • Bidirectional cluster linking is supported when you deploy Confluent for Kubernetes (CFK) 3.2.0 or later with Confluent Platform 7.5 or later. This applies to both ZooKeeper-based and KRaft-based clusters. On earlier CFK or Confluent Platform versions, use unidirectional cluster links, or create and maintain bidirectional cluster links outside of CFK using the Confluent REST API or Confluent CLI.

  • Mirror topics are created and managed in the CFK ClusterLink custom resource (CR).

    Warning

    If you have mirror topics that were created outside of CFK, such as using REST API or Confluent CLI, those mirror topics are not recognized by CFK and are deleted upon reconciliation when CFK restarts next time.

  • It is not possible to set up Cluster Linking with a proxy.

Bidirectional cluster linking

Bidirectional cluster linking allows data to flow in both directions between two Kafka clusters, so that topics can be mirrored from Cluster A to Cluster B and from Cluster B back to Cluster A. This is useful for disaster recovery (DR), active/active deployments, and certain migration patterns where consumer offsets and ACLs must be synchronized in both directions. For an overview of bidirectional mode and supported use cases, see Bidirectional Cluster Linking.

In addition to the general Cluster Linking requirements on this page, make sure that:

  • Each participating Kafka cluster has a password encoder secret configured (spec.passwordEncoder.secretRef) so that sensitive link configuration, such as credentials, can be encrypted.

  • Admin REST is enabled for both clusters, and a KafkaRestClass is configured for each cluster so that CFK can create and manage the links.

How CFK represents a bidirectional link

In CFK, a bidirectional link is represented as two ClusterLink custom resources:

  • One ClusterLink CR handles forward mirroring (for example, topics from a cluster in namespace source to a cluster in namespace destination).

  • A second ClusterLink CR handles reverse mirroring (for example, topics from the cluster in destination back to the cluster in source).

Both ClusterLink CRs must:

  • Use the same logical link name in spec.name. This associates the two CRs as a single bidirectional link pair.

  • Set sourceInitiatedLink.linkMode: Bidirectional to place the underlying cluster link in bidirectional mode.

Types of bidirectional cluster links

There are two types of bidirectional cluster links in CFK:

  • Destination-initiated bidirectional links: Both clusters use standard (outbound) connections. Each cluster can connect to the other directly. This is the simpler setup and works when both clusters are accessible to each other.

  • Source-initiated bidirectional links: Used when one cluster is behind a firewall or in a private network. The private cluster uses an INBOUND connection mode, while the public cluster uses OUTBOUND mode to initiate connections to the private cluster.

Example: Destination-initiated bidirectional link

The following example shows a destination-initiated bidirectional cluster link between two CFK-managed Kafka clusters in namespaces source and destination. One ClusterLink CR mirrors forward-topic from source to destination, and the other mirrors reverse-topic from destination back to source

# Destination-side ClusterLink (receives forward mirrors)
apiVersion: platform.confluent.io/v1beta1
kind: ClusterLink
metadata:
  name: dest-cluster-link
  namespace: destination
spec:
  # Logical link name: must match on both sides of the bidirectional link
  name: bidirectional-link

  # Enable bidirectional mode on the underlying cluster link
  sourceInitiatedLink:
    linkMode: Bidirectional

  # |ak| cluster references
  destinationKafkaCluster:
    name: kafka-dest
    namespace: destination
    # authentication: ...   # required for SASL-SSL clusters
  sourceKafkaCluster:
    name: kafka-src
    namespace: source
    # authentication: ...   # required for SASL-SSL clusters

  # Forward mirror topics: source (source) -> destination (destination)
  mirrorTopics:
    - name: forward-topic
      direction: toDestination

# Source-side ClusterLink (receives reverse mirrors)
apiVersion: platform.confluent.io/v1beta1
kind: ClusterLink
metadata:
  name: src-cluster-link
  namespace: source
spec:
  # Same logical link name as the destination-side ClusterLink
  name: bidirectional-link

  sourceInitiatedLink:
    linkMode: Bidirectional

  # For reverse mirroring, the "source" of data is the destination cluster
  sourceKafkaCluster:
    name: kafka-dest
    namespace: destination
    # authentication: ...   # required for SASL-SSL clusters
  destinationKafkaCluster:
    name: kafka-src
    namespace: source
    # authentication: ...   # required for SASL-SSL clusters

  # Reverse mirror topics: destination (destination) -> source (source)
  mirrorTopics:
    - name: reverse-topic
      direction: toDestination

When configuring destination-initiated bidirectional Cluster Linking with CFK:

  • Both ClusterLink CRs must use the same value for spec.name. If the names differ, the CRs behave as separate unidirectional links instead of a single bidirectional pair.

  • For the reverse direction, sourceKafkaCluster and destinationKafkaCluster are intentionally swapped, because the source of the data for reverse mirroring is the remote cluster.

  • Ensure that each Kafka cluster participating in the link has a working KafkaRestClass and password encoder secret. Otherwise, bidirectional links may fail to create or reconcile.

  • For SASL-SSL secured clusters, configure authentication on both sourceKafkaCluster and destinationKafkaCluster so that CFK can generate both the remote and local JAAS configurations required by the bidirectional links.

Example: Source-initiated bidirectional link

Source-initiated bidirectional links are used when one cluster (typically the destination) is in a private network and cannot accept inbound connections. The source cluster initiates outbound connections to the destination.

Important

Critical ordering requirement: When creating source-initiated bidirectional cluster links, you must create the destination-side ClusterLink CR before creating the source-side ClusterLink CR. If you create them in reverse order, the link will fail to establish.

The following example shows a source-initiated bidirectional cluster link where the destination cluster is in a private network:

# Step 1: Create destination-side ClusterLink FIRST (in private network)
apiVersion: platform.confluent.io/v1beta1
kind: ClusterLink
metadata:
  name: dest-cluster-link
  namespace: destination
spec:
  name: bidirectional-link

  sourceInitiatedLink:
    linkMode: Bidirectional
    connectionMode: Inbound  # Accepts inbound connections from source

  destinationKafkaCluster:
    kafkaRestClassRef:
      name: kafka-dest-rest

  sourceKafkaCluster:
    clusterID: <source-cluster-id>  # Required for INBOUND mode
    kafkaRestClassRef:
      name: kafka-src-rest

  mirrorTopics:
    - name: forward-topic
      direction: toDestination

---
# Step 2: Create source-side ClusterLink AFTER destination (in public network)
apiVersion: platform.confluent.io/v1beta1
kind: ClusterLink
metadata:
  name: src-cluster-link
  namespace: source
spec:
  name: bidirectional-link

  sourceInitiatedLink:
    linkMode: Bidirectional
    connectionMode: Outbound  # Initiates outbound connections

  sourceKafkaCluster:
    bootstrapEndpoint: kafka-dest.destination.svc.cluster.local:9092
    kafkaRestClassRef:
      name: kafka-dest-rest
    authentication:
      type: plain
      jaasConfig:
        secretRef: dest-jaas-secret
    tls:
      enabled: true
      secretRef: dest-tls-secret

  destinationKafkaCluster:
    kafkaRestClassRef:
      name: kafka-src-rest

  mirrorTopics:
    - name: reverse-topic
      direction: toDestination

When configuring source-initiated bidirectional Cluster Linking with CFK:

  • The destination cluster link uses connectionMode: Inbound to accept connections.

  • The source cluster link uses connectionMode: Outbound to initiate connections to the destination.

  • The source-side ClusterLink CR must include full connection details (bootstrapEndpoint, authentication, tls) for reaching the destination cluster.

  • The destination-side ClusterLink CR requires the clusterID of the source cluster when using INBOUND mode.

Additional examples

For complete end-to-end examples of bidirectional cluster linking:

Requirements for destination-initiated cluster linking

  • The destination cluster must be a Confluent Platform or a Confluent Cloud cluster.

  • Cluster links are created and managed using Admin REST APIs, therefore the Admin REST APIs service must be running on the destination cluster. Set up the REST Class as described in Manage Confluent Admin REST Class for Confluent Platform Using Confluent for Kubernetes.

  • The destination Kafka is a consumer to the source Kafka. Therefore, the destination cluster needs to configure authentication/encryption to talk to the source cluster if the source Kafka has security settings enabled. For the required step, see Configure destination Kafka security.

Requirements for source-initiated cluster linking

  • The source cluster and the destination cluster must be a Confluent Platform or a Confluent Cloud cluster.

  • Cluster links are created and managed using Admin REST APIs, therefore the Admin REST service must be running on the source and destination clusters. Set up the Confluent REST Class as described in Manage Confluent Admin REST Class for Confluent Platform Using Confluent for Kubernetes.

  • The source Kafka cluster needs to configure authentication and encryption to communicate with the destination cluster if the destination Kafka has security settings enabled. For the required step, see Configure destination Kafka security.

Create a mirror topic

Mirror topics are created by and owned by a cluster link.

A mirror topic is created as a new topic with a unique name on the destination cluster. There must be a topic of this same name on the cluster link’s source cluster, which will be the mirror topic’s source topic.

You cannot convert an existing read/write topic into a mirror topic.

For details on mirror topic creation in Confluent Platform, see Mirror Topic Creation.

To create a mirror topic:

  1. Set the following in the ClusterLink CR:

    spec:
  mirrorTopics:
    - name:                   --- [1]
      state:                  --- [2]
      configs:                --- [3]
      replicationFactor:      --- [4]
      sourceTopicName:        --- [5]

    • [1] Required. The mirror topic name. If the sourceTopicName is not configured, a topic with the exact same name must exist on the source cluster, and no topic with this name should exist on the destination cluster.

      When spec.mirrorTopicOptions.prefix: <prefix> is configured for the cluster link, the name has to be of the format <prefix><sourceTopicName>.

    • [2] Optional. The state of this mirror topic when created. Possible values are PAUSE, PROMOTE, FAILOVER, ACTVE. If state is not defined, the default value is ACTIVE.

    • [3] Optional. For the list of optional configurations, see Cluster Linking Config Options.

    • [4] Optional. The replication factor for the mirror topic on the destination cluster. If omitted, this mirror topic will inherit the broker’s default replication factor.

    • [5] Optional. The topic name on the source cluster that will be mirrored to the destination cluster.

      When spec.mirrorTopicOptions.prefix is not configured, do not configure this field.

      When spec.mirrorTopicOptions.prefix is configured, a topic with the exact same name must exist on the source cluster.

  2. Apply the changes by running:

    kubectl apply -f <Cluster Link CR>

Modify a mirror topic

You can promote, failover, or pause a mirror topic:

  • Promote a topic: Promote a topic when you want to stop mirroring and make the topic on destination writeable. Verifies if the lag is zero and does final sync before promoting the topic.

  • Failover a topic: This is similar to promote, but doesn’t wait for the lag to become zero.

To learn more about what happens to a Confluent Platform mirror topic when promoted or failed over, see Converting a Mirror Topic to a Normal Topic.

For the available states and statues of a mirror topic in Confluent Platform, see Mirror Topic States and Statuses.

When you promote or failover a mirror topic, CFK creates a KafkaTopic CR. The name of the new KafkaTopic CR will be as below. Note that the topic name in the destination cluster does not change and remains the same as the mirror topic name.

clink-toLowerCase(<topic-name>)-<First section of the ClusterLink object UID>

For example, when a mirror topic, myMirrorTopic, is promoted, the new KafkaTopic CR name is:

clink-mymirrortopic-0154a475

If the new KafkaTopic CR name contains more than 63 characters, it will be truncated to the length of 63 characters long.

To modify an active mirror topic to promote, fail over, or pause:

  1. Modify the mirrorTopics section in the ClusterLink CR:

    spec:
  mirrorTopics:
    - name:                      --- [1]
      state:                     --- [2]

    • [1] Required.

    • [2] Optional. The state of this mirror topic. Possible values are PAUSE, PROMOTE, FAILOVER, ACTIVE.

  2. Apply the changes by running:

    kubectl apply -f clusterlink.yaml

Delete a mirror topic

For the process of deleting a mirror topic in Confluent Platform, see Mirror Topic Deletion.

For the available states and statues of a mirror topic in Confluent Platform, see Mirror Topic States and Statuses.

To delete a mirror topic:

  1. Delete a mirror topic from the list in the ClusterLink CR by removing the name and the other properties for the mirror topic.

    spec:
  mirrorTopics:
    - name:
      state:
      configs:
      replicationFactor:
      sourceTopicName:

  2. Apply the changes by running:

    kubectl apply -f clusterlink.yaml