Cluster Linking with Confluent for Kubernetes Blueprints

The Cluster Linking feature allows you to directly connect Kafka clusters together and mirror topics from one cluster (the source) to another (the destination). You can use the feature in multi-datacenter, multi-cluster, and hybrid cloud deployments to seamlessly move data between Kafka clusters.

Confluent for Kubernetes (CFK) Blueprints provides a declarative API, the ClusterLink custom resource definition (CRD), for defining, updating, and managing cluster links.

The following types of cluster links are supported in CFK Blueprints. Select a type of cluster link based on where you want to move data from and to, and configure the ClusterLink custom resource (CR) accordingly:

Confluent Platform to Confluent Platform cluster link

To move data from an on-prem Confluent Platform to another on-prem Confluent Platform or from Confluent Cloud to Confluent Cloud, create a cluster link on the destination with connection and security information for the destination to talk to the source. No changes are required on the source cluster.

The destination cluster initiates the connection to the source cluster.

You can also use this type of cluster links to migrate data from Apache Kafka® or from other Kafka, such as Amazon MSK, to an on-prem Confluent Platform.

In this document, this type of cluster link is referred to as destination-initiated. Note that this is not a term used in the other cluster linking documentation.

Confluent Platform to Confluent Cloud cluster link

To move data from an on-prem Confluent Platform to Confluent Cloud in a hybrid environment, use the source-initiated type of cluster links.

For source-initiated cluster links, you configure connection and security information on the source cluster to talk to the destination.

The source cluster (on-prem Confluent Platform) initiates the connection to the destination cluster (Confluent Cloud).

For source-initiated cluster links, cluster links need to be configured both on the source and destination clusters.

The high-level workflow to run Cluster Linking is:

  1. Have the source and destination Kafka clusters deployed. For deploying a Kafka using CFK Blueprints, see Deploy and Manage Confluent Platform using Confluent for Kubernetes Blueprints.

    • For a destination-initiated cluster linking, the destination cluster must be a Confluent Platform or a Confluent Cloud cluster.
    • For a source-initiated cluster linking, the source cluster and the destination cluster must be a Confluent Platform or a Confluent Cloud cluster.
  2. If the source or destination Kafka cluster is in Confluent Cloud, set up the connection to the Kafka cluster in Confluent Cloud.

  3. Set up the password encoder for the source and destination Kafka clusters.

    When security is enabled, the source and destination Kafka brokers must be configured with a password encoder secret for encrypting sensitive link configurations, such as certificates and passwords.

    See Updating Password Configurations Dynamically for more about the secret encoder and how to rotate the encoder secret.

  4. Create a cluster link.

Manage mirror topics

Configure mirror topic options

Mirror topic options specify configuration for mirror topics.

To configure mirror topic options for a cluster link, set the following properties in the ClusterLink CR:

apiVersion: apps.cpc.platform.confluent.io/v1beta1
kind: ClusterLink
spec:
  mirrors:
    options:
      autoCreateTopics:  --- [1]
        enabled:         --- [2]
        topicFilters:    --- [3]
          - filterType:  --- [4]
            name:        --- [5]
            patternType: --- [6]
      prefix:            --- [7]
  • [1] Configurations for the cluster link to automatically create mirror topics on the destination cluster. For more info, see Auto-create Mirror Topics.

  • [2] Set to true to automatically create mirror topics on the destination cluster.

    Setting this option to false disables mirror topic creation and clears existing filters.

  • [3] An array of filters to indicate which topics should be mirrored.

  • [4] Required when defining a topic filter. The topic filter type. Valid options are INCLUDE and EXCLUDE.

    If set to INCLUDE, any topic names on the source cluster that match this filter will be created as mirror topics.

    If set to EXCLUDE, any matching topic names will not be created as mirror topics. In other words, it prevents automatically creating mirror topics for the specified topic names.

    EXCLUDE filters override any overlapping INCLUDE filters.

  • [5] Required when defining a topic filter. The resource name associated with this filter. Set the name to the wildcard, *, to mirror all topics.

  • [6] Required when defining a topic filter. The pattern of the resource. Valid options are PREFIXED and LITERAL.

    If the name is set to foo, then setting patternType to LITERAL will only match a topic named foo. Setting patternType to PREFIXED will match any topic names that begin with foo, for example, foo, football, and foo.fighters.

  • [7] The prefix for mirror topics. When auto-create is enabled (spec.mirrors.options.autoCreateTopics.enabled: true),and the prefix is configured, the topics created on the destination cluster will automatically contain the prefix. Otherwise, spec.mirrorTopic.name must be defined in the <prefix><sourceTopicName> format.

Create a mirror topic

Mirror topics are created by and owned by a cluster link.

A mirror topic is created as a new topic with a unique name on the destination cluster. There must be a topic of this same name on the cluster link’s source cluster, which will be the mirror topic’s source topic.

You cannot convert an existing read/write topic into a mirror topic.

For details on mirror topic creation in Confluent Platform, see Mirror Topic Creation.

To create a mirror topic:

  1. Set the following in the ClusterLink CR and apply the changes using the kubectl apply command:

    apiVersion: apps.cpc.platform.confluent.io/v1beta1
    kind: ClusterLink
    spec:
      mirrors:
        topics:
          - name:                   --- [1]
            state:                  --- [2]
            configs:                --- [3]
            replicationFactor:      --- [4]
            sourceTopicName:        --- [5]
    
    • [1] Required. The mirror topic name. If the sourceTopicName is not configured, a topic with the exact same name must exist on the source cluster, and no topic with this name should exist on the destination cluster.

      When spec.mirrors.options.prefix: <prefix> is configured for the cluster link, the name has to be of the format <prefix><sourceTopicName>.

    • [2] Optional. The state of this mirror topic when created. Possible values are PAUSE, PROMOTE, FAILOVER, ACTVE. If the state is not defined, the default value is ACTIVE.

    • [3] Optional. For the list of optional configurations, see Cluster Linking Config Options.

    • [4] Optional. The replication factor for the mirror topic on the destination cluster. If omitted, this mirror topic will inherit the broker’s default replication factor.

    • [5] Optional. The topic name on the source cluster that will be mirrored to the destination cluster.

      When spec.mirrors.options.prefix is not configured, do not configure this field.

      When spec.mirrors.options.prefix is configured, a topic with the exact same name must exist on the source cluster.

Modify a mirror topic

You can promote, failover, or pause a mirror topic:

  • Promote a topic: Promote a topic when you want to stop mirroring and make the topic on destination writeable. Verifies if the lag is zero and does final sync before promoting the topic.
  • Failover a topic: This is similar to promoting, but doesn’t wait for the lag to become zero.

To modify an active mirror topic to promote, failover, or pause:

  1. Modify the mirrorTopics section in the ClusterLink CR and apply the changes with the kubectl command:

    apiVersion: apps.cpc.platform.confluent.io/v1beta1
    kind: ClusterLink
    spec:
      mirrors:
        topics:
          - name:                      --- [1]
            state:                     --- [2]
    
    • [1] Required.
    • [2] Optional. The state of this mirror topic. Possible values are PAUSE, PROMOTE, FAILOVER, ACTIVE.

Delete a mirror topic

For the process of deleting a mirror topic in Confluent Platform, see Mirror Topic Deletion.

For the available states and status of a mirror topic in Confluent Platform, see Mirror Topic States and Statuses.

To delete a mirror topic:

  1. Delete a mirror topic from the list in the ClusterLink CR by removing the name and the other properties for the mirror topic. And apply the changes with the kubectl apply command.

    apiVersion: apps.cpc.platform.confluent.io/v1beta1
    kind: ClusterLink
    spec:
      mirrors:
        topics:
          - name:
            state:
            configs:
            replicationFactor:
            sourceTopicName: