Cluster Linking using Confluent for Kubernetes

Using the Cluster Linking feature, you can directly connect Kafka clusters together and mirror topics from one cluster (the source) to another (the destination). The feature makes it easier for you to build multi-datacenter, multi-cluster, and hybrid cloud deployments.

Confluent for Kubernetes (CFK) provides a declarative API, the ClusterLink custom resource definition (CRD), for defining, updating, and managing cluster links.

Cluster link configurations are done on the destination cluster. No changes are required on the source cluster.

Requirements

Before you create cluster links, make sure the following requirements are satisfied:

  • The destination cluster must be a Confluent Platform or Confluent Cloud cluster.
  • Cluster links are created and managed using Admin REST APIs, and Admin REST APIs must be running on the destination cluster. If your destination cluster is managed by CFK, instructions for setting up Admin REST APIs are described in Confluent Admin REST API.
  • The destination Kafka is a consumer to the source Kafka. Therefore, the destination cluster needs to configure authentication/encryption to talk to the source cluster if the source Kafka has security settings enabled. For the required step, see Configure destination Kafka security.

Create a mirror topic

Mirror topics are created by and owned by a cluster link.

A mirror topic is created as a new topic with a unique name on the destination cluster. There must be a topic of this same name on the cluster link’s source cluster, which will be the mirror topic’s source topic.

You cannot convert an existing read/write topic into a mirror topic.

To create a mirror topic:

  1. Set the following in the ClusterLink CR:
spec:
  mirrorTopics:
    - name:                      --- [1]
      state:                     --- [2]
      configs:                   --- [3]
  • [1] Required. The name of the source topic. There must be a topic of this same name on the cluster link’s source cluster.
  • [2] Optional. The state of this mirror topic when created. Possible values are PAUSE, PROMOTE, FAILOVER, ACTVE. If state is not defined, the default value is ACTIVE.
  • [3] Optional. For the list of optional configurations, see Cluster Linking Config Options.
  1. Apply the changes by running:

    kubectl apply -f <Cluster Link CR>
    

Modify a mirror topic

You can promote, failover, or pause a mirror topic:

  • Promote a topic: Promote a topic when you want to stop the mirror and make the topic on destination writeable. Verifies if the lag is zero and does final sync before promoting the topic.
  • Failover a topic: This is similar to promote, but doesn’t wait for the lag to become zero.

When you promote or failover a mirror topic, CFK creates a KafkaTopic CR. The name of the new Kafka topic will be:

clink-toLowerCase(<topic-name>)-<First section of the ClusterLink object UID>

For example, when a mirror topic, myMirrorTopic, is promoted, the new name is:

clink-mymirrortopic-0154a475

If the new topic name contains more than 63 characters, it will be truncated to the length of 63 characters long.

Modify an active mirror topic to promote, fail over, or pause:

  1. Modify the mirrorTopics section in the ClusterLink CR:

    spec:
      mirrorTopics:
        - name:                      --- [1]
          state:                     --- [2]
    
    • [1] Required.
    • [2] Optional. The state of this mirror topic. Possible values are PAUSE, PROMOTE, FAILOVER, ACTIVE.
  2. Apply the changes by running:

    kubectl apply -f clusterlink.yaml
    

Delete a mirror topic

To delete a mirror topic:

  1. Remove the mirror topic from the list in the ClusterLink CR:

    spec:
      mirrorTopics:
        - name:                      --- [1]
    
    • [1] Remove the name and the other properties for the mirror topic.
  2. Apply the changes by running:

    kubectl apply -f clusterlink.yaml