1. Home
2. Cluster Linking

Looking for Confluent Platform Cluster Linking docs? This page describes Cluster Linking on Confluent Cloud. If you are looking for Confluent Platform documentation, check out Cluster Linking on Confluent Platform.

Cluster Linking Configuration, Commands, and Management

You can create, view, manage, and delete cluster links using the unified Confluent CLI and the Confluent Cloud REST API.

Note

The Cluster Linking REST commands to create and update cluster links are available on all Confluent Cloud Dedicated Confluent Cloud clusters with Internet networking created on or after August 17, 2021. They are being rolled out to all previously created clusters. To request that your existing cluster be prioritized in this roll out, send an email to clusterlinking@confluent.io.

Requirements to Create a Cluster Link

To create a cluster link, you will need:

• Access to the Destination cluster. Cluster links are created on the destination cluster. For the specific security authorization needed, see Security for Cluster Linking on Confluent Cloud.
• The name you wish to give the cluster link.
• The Source cluster’s bootstrap server and cluster ID.
  • If the Source cluster is a Confluent Cloud cluster, you can get those using these commands:
    • confluent kafka cluster list to get the Source cluster’s ID. For example, lkc-12345
    • confluent kafka cluster describe <source-cluster-id> to get the bootstrap server. It will be the value of the row called Endpoint. For example, SASL_SSL://pkc-12345.us-west-2.aws.confluent.cloud:9092
  • If the Source cluster is a Confluent Platform or Apache Kafka® cluster, your system administrator can give you your bootstrap server. You can get your cluster ID by doing confluent cluster describe or zookeeper-shell <bootstrap-server> get /cluster/id 2> /dev/null. For example, 0gXf5xg8Sp2G4FlxNriNaA.
  • Authentication credentials for the cluster link to use with the Source cluster. These can be updated, allowing you to rotate credentials used by your cluster link. See the section on configuring security credentials for more details.
• (Optional) Any additional configuration parameters you wish to add to the cluster link. See the list of possible configurations below.

You will then pass this information into the CLI or API command to create a cluster link.

Note

You cannot update the cluster link’s name, bootstrap server, Source cluster ID, or Destination cluster ID. These values are only set at creation.

Managing Cluster Links with the CLI

To create or configure a cluster link with the Confluent Cloud CLI, you will need to create a config file with the configurations you wish to set. Each configuration should be on one line in the file, with the format key=value, where key is the name of the config, and value is the value you wish to set on that cluster link. In the following commands, the location of this file is referred to as <config-file>.

To create a cluster link with the confluent CLI, use this command on the Destination cluster:

confluent kafka link create <link-name> \
  --source-cluster-id <source-cluster-id> \
  --source-bootstrap-server <source-bootstrap-server> \
  --config-file <config-file> \
  --cluster <destination-cluster-id>

To update the configuration for an existing cluster link with the cloud CLI, use this command on the Destination cluster:

confluent kafka link update <link-name>  --cluster <destination-cluster-id>

Tip

• Examples of creating and configuring cluster links with the CLI are shown in the Cluster Linking Quick Start, and the tutorials for Share Data Across Clusters, Regions, and Clouds and Disaster Recovery and Failover.
• You can change several aspects of a cluster link configuration, but you cannot change its source cluster (bootstrap servers, ID, and so on) or the link name.

To see a list of all cluster links going to a Destination cluster, use the command:

confluent kafka link list --cluster <destination-cluster-id>

To view the configuration of a specific cluster link , use the command:

confluent kafka link describe <link-name> --cluster <destination-cluster-id>

To delete a cluster link, use the command:

confluent kafka link delete <link-name> --cluster <destination-cluster-id>

Tip

Using the command confluent kafka cluster use <destination-cluster-id> sets the Destination cluster to the active cluster, so you won’t need to specify --cluster <destination-cluster-id> in any commands.

Note

When a cluster link is deleted, so is the history of any STOPPED topics. If you need the Last Source Fetch Offset or the Status Time of your promoted or failed-over mirror topics, make sure to save those before you delete the cluster link.

Managing Cluster Links with the REST API

Tip

To learn how to authenticate with the REST API, see the REST API documentation.

To use the Confluent Cloud REST API to configure cluster links, you can pass in configs in JSON format. Each config will take this shape:

{
  "name": "<config-name>",
  "value": <config-value>
}

Since this is JSON, if your config value is a string, surround it in double quotes. Then, pass in the name/value pairs as an array to the configs argument of the JSON payload.

For example, this JSON enables consumer offset sync for all consumer groups and ACL sync for all ACLs.

{
  "configs": [
    {
        "name": "consumer.offset.sync.enable",
        "value": "true"
    },
    {
        "name": "consumer.offset.sync.filters",
        "value": "{\"groupFilters\": [{\"name\": \"*\",\"patternType\": \"LITERAL\",\"filterType\": \"INCLUDE\"}]}"
    },
    {
        "name": "acl.sync.enable",
        "value": "true"
    },
    {
        "name": "acl.filters",
        "value": "{ \"aclFilters\": [ { \"resourceFilter\": { \"resourceType\": \"any\", \"patternType\": \"any\" }, \"accessFilter\": { \"operation\": \"any\", \"permissionType\": \"any\" } } ] }"
    }
  ]
}

Creating a Cluster Link through the REST API

To create a cluster link using the REST API, the JSON payload needs:

• an entry called source_cluster_id set to your Source cluster’s ID
• a config in the configs array called bootstrap.servers set to your Source cluster’s bootstrap server
• your cluster link’s security configuration listed as configs in the configs array
• (optional) additional configurations in the configs array.

Example JSON payload for a request to create a cluster link that syncs consumers offsets for all consumer groups, and syncs all ACLs:

{
  "source_cluster_id": "<source-cluster-id>",
  "configs": [
    {
      "name": "security.protocol",
      "value": "SASL_SSL"
    },
    {
      "name": "bootstrap.servers",
      "value": "<source-bootstrap-server>"
    },
    {
      "name": "sasl.mechanism",
      "value": "PLAIN"
    },
    {
      "name": "sasl.jaas.config",
      "value": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"<source-api-key>\" password=\"<source-api-secret>\";"
    }
  ]
}

Include this JSON payload in a POST request to:

<REST-Endpoint>/kafka/v3/clusters/<cluster-ID>/links/?link_name=<link-name>

Where you replace the following:

• <REST-Endpoint> is your cluster’s REST url. You can find this with confluent kafka cluster describe <cluster-ID>. It wil look like: https://pkc-XXXXX.us-west1.gcp.confluent.cloud:443
• <cluster-ID> is your destination cluster’s ID. You can find this with confluent kafka cluster list. It will look like lkc-XXXXX.
• <link-name> is the name of your cluster link. You can name it whatever you choose.

Updating and Viewing Cluster Link Configs with the REST API

To update an existing link, you will send the JSON payload described above as a PUT request to <REST-Endpoint>/clusters/<cluster-ID>/links/<link-name>/configs:alter

To get the value of your cluster link’s configurations using the REST API, use one of these options:

• GET <REST-Endpoint>/clusters/<cluster-ID>/links/<link-name>/configs to get the full list of configs.
• GET <REST-Endpoint>/clusters/<cluster-ID>/links/<link-name>/<config-name> to get the value of a specific config.

To view a list of all cluster links going to a Destination cluster, send a GET request to <REST-Endpoint>/clusters/<cluster-ID>/links.

To delete a cluster link, send a DELETE request to <REST-Endpoint>/clusters/<cluster-ID>/links/<link-name>.

Note

When a cluster link is deleted, so is the history of any STOPPED topics. If you need the Last Source Fetch Offset or the Status Time of your promoted or failed-over mirror topics, make sure to save those before you delete the cluster link.

Configuring Cluster Link Behavior

These properties are available to control the behavior of a cluster link that goes to a Confluent Cloud destination cluster.

acl.filters

JSON string that lists the ACLs to migrate. Define the ACLs in a file, acl.filters.json, and pass the file name as an argument to --acl-filters-json-file.

• Type: string
• Default: “”

Note

• Populate acl.filters by passing a JSON file on the command line that specifies the ACLs.
• Do not include in the filter ACLs that are managed independently on the destination cluster. This is to prevent cluster link migration from deleting ACLs that were added specifically on the destination and should not be deleted. See also, ACL Syncing.

acl.sync.enable

Whether or not to migrate ACLs. See also, ACL Syncing.

• Type: boolean
• Default: false

acl.sync.ms

How often to refresh the ACLs, in milliseconds (if ACL migration is enabled). The default is 5000 milliseconds (5 seconds).

• Type: int
• Default: 5000

consumer.offset.group.filters

JSON to denote the list of consumer groups to be migrated. To learn more, see Migrating Consumer Groups from Source to Destination Cluster.

• Type: string
• Default: “”

consumer.offset.sync.enable

Whether or not to migrate consumer offsets from the source cluster.

• Type: boolean
• Default: false

consumer.offset.sync.ms

How often to sync consumer offsets, in milliseconds, if enabled.

• Type: int
• Default: 30000

topic.config.sync.ms

How often to refresh the topic configs, in milliseconds.

• Type: int
• Default: 5000

Configuring Security Credentials

When you create a cluster link, you will need to give the cluster link credentials with which to authenticate into the Source cluster. Read the Cluster Linking security overview for more information about Cluster Linking security requirements, including the required ACLs.

Configuring Security Credentials for Confluent Cloud Source Clusters

If your Source cluster is a Confluent Cloud cluster, your cluster link will need these properties configured:

• security.protocol set to SASL_SSL
• sasl.mechanism set to PLAIN
• sasl.jaas.config set to org.apache.kafka.common.security.plain.PlainLoginModule required username="<source-api-key>" password="<source-api-secret>";
  • Replace <source-api-key> and <source-api-secret> with the API key to use on your Source cluster.
  • Be careful not to forget the " characters and the ending ;

Tip

When using the confluent CLI in testing and development, you can use the parameters ``

Configuring Security Credentials for Confluent Platform and Kafka Source Clusters

Since cluster links consume from the source cluster using the same mechanism as a regular Kafka consumer, the cluster link’s authentication configuration is identical to the one needed by Kafka consumers.

The following properties are supported by cluster links going to Confluent Destination clusters:

• security.protocol
• sasl.mechanism
• sasl.jaas.config
• ssl.cipher.suites
• ssl.protocol
• ssl.enabled.protocols
• ssl.endpoint.identification.algorithm

Suggested Reading

• Unified Confluent CLI reference
• Confluent Cloud REST API
• Security for Cluster Linking on Confluent Cloud
• Tutorials