Cluster Linking using Confluent for Kubernetes

Using the Cluster Linking feature, you can directly connect Kafka clusters together and mirror topics from one cluster (the source) to another (the destination). The feature makes it easier for you to build multi-datacenter, multi-cluster, and hybrid cloud deployments.

Confluent for Kubernetes (CFK) provides a declarative API, the ClusterLink custom resource definition (CRD), for defining, updating, and managing cluster links.

Cluster link configurations are done on the destination cluster. No changes are required on the source cluster.

Requirements

Before you create cluster links, make sure the following requirements are satisfied:

  • The destination cluster must be a Confluent Platform or Confluent Cloud cluster.
  • Cluster links are created and managed using Admin REST APIs, and Admin REST APIs must be running on the destination cluster. If your destination cluster is managed by CFK, instructions for setting up Admin REST APIs are described in Confluent Admin REST API.
  • The destination Kafka is a consumer to the source Kafka. Therefore, the destination cluster needs to configure authentication/encryption to talk to the source cluster if the source Kafka has security settings enabled. For the required step, see Configure destination Kafka security.

Create a mirror topic

Mirror topics are created by and owned by a cluster link.

A mirror topic is created as a new topic with a unique name on the destination cluster. There must be a topic of this same name on the cluster link’s source cluster, which will be the mirror topic’s source topic.

You cannot convert an existing read/write topic into a mirror topic.

For details on mirror topic creation in Confluent Platform, see Mirror Topic Creation.

To create a mirror topic:

  1. Set the following in the ClusterLink CR:

    spec:
      mirrorTopics:
        - name:                      --- [1]
          state:                     --- [2]
          configs:                   --- [3]
    
    • [1] Required. The name of the source topic. There must be a topic of this same name on the cluster link’s source cluster.
    • [2] Optional. The state of this mirror topic when created. Possible values are PAUSE, PROMOTE, FAILOVER, ACTVE. If state is not defined, the default value is ACTIVE.
    • [3] Optional. For the list of optional configurations, see Cluster Linking Config Options.
  2. Apply the changes by running:

    kubectl apply -f <Cluster Link CR>
    

Modify a mirror topic

You can promote, failover, or pause a mirror topic:

  • Promote a topic: Promote a topic when you want to stop mirroring and make the topic on destination writeable. Verifies if the lag is zero and does final sync before promoting the topic.
  • Failover a topic: This is similar to promote, but doesn’t wait for the lag to become zero.

To learn more about what happens to a Confluent Platform mirror topic when promoted or failed over, see Converting a Mirror Topic to a Normal Topic.

For the available states and statues of a mirror topic in Confluent Platform, see Mirror Topic States and Statuses.

When you promote or failover a mirror topic, CFK creates a KafkaTopic CR. The name of the new KafkaTopic CR will be as below. Note that the topic name in the destination cluster does not change and remains the same as the mirror topic name.

clink-toLowerCase(<topic-name>)-<First section of the ClusterLink object UID>

For example, when a mirror topic, myMirrorTopic, is promoted, the new KafkaTopic CR name is:

clink-mymirrortopic-0154a475

If the new KafkaTopic CR name contains more than 63 characters, it will be truncated to the length of 63 characters long.

To modify an active mirror topic to promote, fail over, or pause:

  1. Modify the mirrorTopics section in the ClusterLink CR:

    spec:
      mirrorTopics:
        - name:                      --- [1]
          state:                     --- [2]
    
    • [1] Required.
    • [2] Optional. The state of this mirror topic. Possible values are PAUSE, PROMOTE, FAILOVER, ACTIVE.
  2. Apply the changes by running:

    kubectl apply -f clusterlink.yaml
    

Delete a mirror topic

For the process of deleting a mirror topic in Confluent Platform, see Mirror Topic Deletion.

For the available states and statues of a mirror topic in Confluent Platform, see Mirror Topic States and Statuses.

To delete a mirror topic:

  1. Remove the mirror topic from the list in the ClusterLink CR:

    spec:
      mirrorTopics:
        - name:                      --- [1]
    
    • [1] Remove the name and the other properties for the mirror topic.
  2. Apply the changes by running:

    kubectl apply -f clusterlink.yaml