1. Home
  2. Cluster Linking

Cluster Linking Configuration, Commands, and Management

Looking for Confluent Platform Cluster Linking docs? You are currently viewing Confluent Cloud documentation. If you are looking for Confluent Platform docs, check out Cluster Linking on Confluent Platform.

You can create, view, manage, and delete cluster links using the unified Confluent CLI and the Confluent Cloud Cluster Linking (v3) REST API.

Syncing ACLs from Source to Destination Cluster

Cluster Linking can sync some or all of the ACLs on the source cluster to the destination cluster. This is critical for Disaster Recovery (DR) architectures, so that when your client applications failover to the DR cluster, they have the same access that they had on their original cluster. When you create the cluster link, you can specify which ACLs to sync, filtering down by the name of the resource, principal (service account), operation, and/or permission. You can update this setting on the cluster link at any time. The cluster link will copy the initial set of matching ACLs, and then continuously sync any changes, additions, or deletions of matching ACLs from the source cluster to the destination cluster.

Tip

You can also migrate consumer group offsets from the source to destination cluster.

Prerequisites

You must have the appropriate authorizations:

  • DESCRIBE Cluster ACLs (DescribeAcls API) on the source cluster
  • ALTER Cluster ACLs (CreateAcls/DeleteAcls APIs) on the destination cluster

Configurations for ACL Sync

To enable ACL sync on a cluster link, specify these properties:

acl.sync.enable=true

This turns on ACL sync. Updating this config to false will turn off ACL sync.

If you set this up and run Cluster Linking, then later disable it, the filters will be cleared (deleted) from the cluster link.

acl.filters
A JSON object with one property, aclFilters, that contains an array of filters to include specific ACLs to sync. Examples are below.
acl.sync.ms (optional)
How frequently to sync ACLs. The default is 5000 (5 seconds). Minimum is 1000.

You can configure ACLs at the time you create the cluster link, or as an update to an existing configuration.

Here is an example of setting up a cluster link with ACL migration when you create the cluster link. Note that the link configurations (including ACL migration properties) are defined in a file, link-configs.properties, rather than specified directly on the command line.

confluent kafka link create from-aws-basic \
    --source-cluster-id lkc-12345 \
    --source-bootstrap-server SASL_SSL://pkc-abcde.us-west1.gcp.confluent.cloud:9092 \
    --source-api-key 1L1NK2RUL3TH3M4LL \
    --source-api-secret ******* \
    --config-file link-configs.properties

Here is an example of what you might have in link-configs.properties:

acl.sync.enable=true
acl.sync.ms=1000
acl.filters={ "aclFilters": [ { "resourceFilter": { "resourceType": "any", "patternType": "any" }, "accessFilter": { "operation": "any", "permissionType": "any" } } ] }

The following sections provide examples of how you might define the actual ACLs in acls.filters.

Examples

The following examples show how to configure the JSON for various types of ACL migration, with granularity on a topic, a resource, or a mixed set.

Migrate all ACLs from source to destination cluster

To migrate all ACLs from the source to the destination cluster, provide the following for acl.filters.

acl.filters={                   \
  "aclFilters": [               \
    {                           \
      "resourceFilter": {       \
        "resourceType": "any",  \
        "patternType": "any"    \
      },                        \
      "accessFilter": {         \
        "operation": "any",     \
        "permissionType": "any" \
      }                         \
    }                           \
  ]                             \
}

Each field in the JSON has the following options:

  • resourceType: Can be ANY (meaning any kind of resource) or topic, cluster, group, or transactionalID, which are the resources specified in Use Access Control Lists (ACLs) for Confluent Cloud.
  • patternType: Can be ANY, LITERAL, PREFIXED, or MATCH.
  • name: Name of resource.
    • If left empty, will default to the * wildcard, which will match all names of the specified resourceType.
    • If patternType is "LITERAL" then any resources with this exact name will be included.
    • If patternType is "PREFIXED", then any resources with a name that has this as a prefix will be included.
    • If patternType is "MATCH", then any resources that match a given pattern will be included. For example, a literal, wildcard (*), or prefixed pattern of the same name and type.
  • principal: (optional) Name of principal specified on the ACL. If left empty will default to the * wildcard, which will match all principals with the specified operation and permissionType.
    • For example, a service account with ID 12345 has this principal: User:sa-12345
  • operation: Can either be any or one of those specified in Operations under the “Operation” column.
  • permissionType: Can be any, allow, or deny

Sync all ACLs specific to a topic

To sync all ACLs specific to a single topic (in this case topic pineapple), provide the following configurations in acls.filters:

acl.filters={                     \
  "aclFilters": [                 \
    {                             \
      "resourceFilter": {         \
        "resourceType": "topic",  \
        "patternType": "literal", \
        "name": "pineapple”       \
      },                          \
      "accessFilter": {           \
        "operation": "any",       \
        "permissionType": "any"   \
      }                           \
    }                             \
   ]                              \
 }

Sync ACLs specific to a principal or permission

To sync all ACLs specific to a service account with ID 12345 and also all ACLs that deny access to the cluster, provide the following configurations in acls.filters:

acl.filters={                         \
  "aclFilters": [                     \
    {                                 \
      "resourceFilter": {             \
        "resourceType": "any",        \
        "patternType": "any"          \
      },                              \
      "accessFilter": {               \
        “principal”: “User:sa-12345”, \
        "operation": "any",           \
        "permissionType": "any"       \
      }                               \
    },                                \
    {                                 \
      "resourceFilter": {             \
        "resourceType": "any",        \
        "patternType": "any"          \
      },                              \
      "accessFilter": {               \
        "operation": "any",           \
        "permissionType": "deny"      \
      }                               \
     }                                \
   ]                                  \
 }

Configuring Security Credentials

When you create a cluster link, you will need to give the cluster link credentials with which to authenticate into the Source cluster. Read the Cluster Linking security overview for more information about Cluster Linking security requirements, including the required ACLs.

Configuring Security Credentials for Confluent Cloud Source Clusters

If your Source cluster is a Confluent Cloud cluster, your cluster link will need these properties configured:

  • security.protocol set to SASL_SSL
  • sasl.mechanism set to PLAIN
  • sasl.jaas.config set to org.apache.kafka.common.security.plain.PlainLoginModule required username="<source-api-key>" password="<source-api-secret>";
    • Replace <source-api-key> and <source-api-secret> with the API key to use on your Source cluster.
    • Be careful not to forget the " characters and the ending ;

Configuring Security Credentials for Confluent Platform and Kafka Source Clusters

Since cluster links consume from the source cluster using the same mechanism as a regular Kafka consumer, the cluster link’s authentication configuration is identical to the one needed by Kafka consumers.

Note

For Confluent Cloud cluster links, if your source cluster uses SSL (also known as TLS) and you need to pass a keystore and truststore; you must do so in-line, rather than with a keystore and truststore location. To learn more, see the security section on Mutual TLS (mTLS).

The following properties are supported by cluster links going to Confluent Destination clusters:

  • sasl.client.callback.handler.class
  • sasl.jaas.config
  • sasl.kerberos.kinit.cmd
  • sasl.kerberos.min.time.before.relogin
  • sasl.kerberos.service.name
  • sasl.kerberos.ticket.renew.jitter
  • sasl.kerberos.ticket.renew.window.factor
  • sasl.login.callback.handler.class
  • sasl.login.class
  • sasl.login.refresh.buffer.seconds
  • sasl.login.refresh.min.period.seconds
  • sasl.login.refresh.window.factor
  • sasl.login.refresh.window.jitter
  • sasl.mechanism
  • security.protocol
  • ssl.cipher.suites
  • ssl.enabled.protocols
  • ssl.endpoint.identification.algorithm
  • ssl.engine.factory.class
  • ssl.key.password
  • ssl.keymanager.algorithm
  • ssl.keystore.location
  • ssl.keystore.password
  • ssl.keystore.type
  • ssl.keystore.certificate.chain
  • ssl.protocol
  • ssl.provider
  • ssl.truststore.certificates