Cluster Linking using Confluent for Kubernetes

The Cluster Linking feature allows you to directly connect Kafka clusters together and mirror topics from one cluster (the source) to another (the destination). You can use the feature in multi-datacenter, multi-cluster, and hybrid cloud deployments to seamlessly move data between Kafka clusters.

Confluent for Kubernetes (CFK) provides a declarative API, the ClusterLink custom resource definition (CRD), for defining, updating, and managing cluster links.

For use case scenarios for configuring cluster links in CFK, refer to Cluster Linking Use Cases in the CFK example GitHub repo.

The following types of cluster links are supported in CFK. Select a type of cluster links based on where you want to move data to and from, and configure the ClusterLink CR accordingly:

  • Confluent Platform to Confluent Platform cluster links

    To move data from an on-prem Confluent Platform to another on-prem Confluent Platform or from Confluent Cloud to Confluent Cloud, create a cluster link on the destination with connection and security information for the destination to talk to the source. No changes are required on the source cluster.

    The destination cluster initiates the connection to the source cluster.

    You can also use this type of cluster links to migrate data from Apache Kafka® or from other Kafka, such as Amazon MSK, to on-prem Confluent Platform.

    In this document, this type of cluster link is referred to as destination-initiated. Note that this is not a term used in the other cluster linking documentation.

  • Confluent Platform to Confluent Cloud cluster links

    To move data from an on-prem Confluent Platform to Confluent Cloud in a hybrid environment, use the source-initiated type of cluster links.

    For source-initiated cluster links, you configure connection and security information on the source cluster to talk to the destination.

    The source cluster (on-prem Confluent Platform) initiates the connection to the destination cluster (Confluent Cloud).

    For source-initiated cluster links, cluster links need to be configured both on the source and destination clusters.

For information on how to create each type of cluster link, see Create a cluster link.

Requirements

Before a cluster link is created, the following requirements must be satisfied:

Requirements for destination-initiated cluster linking

  • The destination cluster must be a Confluent Platform or a Confluent Cloud cluster.
  • Cluster links are created and managed using Admin REST APIs, therefore the Admin REST APIs service must be running on the destination cluster. Set up the REST Class as described in Manage Confluent Admin REST Class using Confluent for Kubernetes.
  • The destination Kafka is a consumer to the source Kafka. Therefore, the destination cluster needs to configure authentication/encryption to talk to the source cluster if the source Kafka has security settings enabled. For the required step, see Configure destination Kafka security.

Requirements for source-initiated cluster linking

  • The source cluster and the destination cluster must be a Confluent Platform or a Confluent Cloud cluster.
  • Cluster links are created and managed using Admin REST APIs, therefore the Admin REST service must be running on the source and destination clusters. Set up the Confluent REST Class as described in Manage Confluent Admin REST Class using Confluent for Kubernetes.
  • The source Kafka cluster needs to configure authentication and encryption to communicate with the destination cluster if the destination Kafka has security settings enabled. For the required step, see Configure destination Kafka security.

Create a mirror topic

Mirror topics are created by and owned by a cluster link.

A mirror topic is created as a new topic with a unique name on the destination cluster. There must be a topic of this same name on the cluster link’s source cluster, which will be the mirror topic’s source topic.

You cannot convert an existing read/write topic into a mirror topic.

For details on mirror topic creation in Confluent Platform, see Mirror Topic Creation.

To create a mirror topic:

  1. Set the following in the ClusterLink CR:

    spec:
  mirrorTopics:
    - name:                   --- [1]
      state:                  --- [2]
      configs:                --- [3]
      replicationFactor:      --- [4]
      sourceTopicName:        --- [5]

    • [1] Required. The mirror topic name. If the sourceTopicName is not configured, a topic with the exact same name must exist on the source cluster, and no topic with this name should exist on the destination cluster.

      When spec.mirrorTopicOptions.prefix: <prefix> is configured for the cluster link, the name has to be of the format <prefix><sourceTopicName>.

    • [2] Optional. The state of this mirror topic when created. Possible values are PAUSE, PROMOTE, FAILOVER, ACTVE. If state is not defined, the default value is ACTIVE.

    • [3] Optional. For the list of optional configurations, see Cluster Linking Config Options.

    • [4] Optional. The replication factor for the mirror topic on the destination cluster. If omitted, this mirror topic will inherit the broker’s default replication factor.

    • [5] Optional. The topic name on the source cluster that will be mirrored to the destination cluster.

      When spec.mirrorTopicOptions.prefix is not configured, do not configure this field.

      When spec.mirrorTopicOptions.prefix is configured, a topic with the exact same name must exist on the source cluster.

  2. Apply the changes by running:

    kubectl apply -f <Cluster Link CR>

Modify a mirror topic

You can promote, failover, or pause a mirror topic:

  • Promote a topic: Promote a topic when you want to stop mirroring and make the topic on destination writeable. Verifies if the lag is zero and does final sync before promoting the topic.
  • Failover a topic: This is similar to promote, but doesn’t wait for the lag to become zero.

To learn more about what happens to a Confluent Platform mirror topic when promoted or failed over, see Converting a Mirror Topic to a Normal Topic.

For the available states and statues of a mirror topic in Confluent Platform, see Mirror Topic States and Statuses.

When you promote or failover a mirror topic, CFK creates a KafkaTopic CR. The name of the new KafkaTopic CR will be as below. Note that the topic name in the destination cluster does not change and remains the same as the mirror topic name.

clink-toLowerCase(<topic-name>)-<First section of the ClusterLink object UID>

For example, when a mirror topic, myMirrorTopic, is promoted, the new KafkaTopic CR name is:

clink-mymirrortopic-0154a475

If the new KafkaTopic CR name contains more than 63 characters, it will be truncated to the length of 63 characters long.

To modify an active mirror topic to promote, fail over, or pause:

  1. Modify the mirrorTopics section in the ClusterLink CR:

    spec:
  mirrorTopics:
    - name:                      --- [1]
      state:                     --- [2]
    • [1] Required.
    • [2] Optional. The state of this mirror topic. Possible values are PAUSE, PROMOTE, FAILOVER, ACTIVE.

  2. Apply the changes by running:

    kubectl apply -f clusterlink.yaml

Delete a mirror topic

For the process of deleting a mirror topic in Confluent Platform, see Mirror Topic Deletion.

For the available states and statues of a mirror topic in Confluent Platform, see Mirror Topic States and Statuses.

To delete a mirror topic:

  1. Delete a mirror topic from the list in the ClusterLink CR by removing the name and the other properties for the mirror topic.

    spec:
  mirrorTopics:
    - name:
      state:
      configs:
      replicationFactor:
      sourceTopicName:

  2. Apply the changes by running:

    kubectl apply -f clusterlink.yaml