Security for Cluster Linking on Confluent Cloud

Looking for Confluent Platform Cluster Linking docs? You are currently viewing Confluent Cloud documentation. If you are looking for Confluent Platform docs, check out Cluster Linking on Confluent Platform.

RBAC Roles and Kafka ACLs Summary

A user needs role-based access control (RBAC) or Kafka ACLs on the destination cluster in order to create and manage cluster links and mirror topics.

The cluster link itself needs specific RBAC roles or Kafka ACLs on the source cluster to read data and metadata.

Note that RBAC role(s) and Kafka ACL(s) are not mutually exclusive. A given user or service account may have a mix of RBAC roles and Kafka ACLs, if that would better suit your security model.

Principal Operation Cluster Kafka RBAC Roles or Kafka ACLs Authentication Options
User, or your tooling Create or modify a cluster link Destination cluster
  • CloudClusterAdmin, or
  • EnvAdmin, or
  • OrgAdmin
or ALTER: Cluster API key or OAuth
User, or your tooling Create or modify a mirror topic Destination cluster
  • CloudClusterAdmin, or
  • EnvAdmin, or
  • OrgAdmin
or
  • ALTER: Cluster, and
  • ALTERCONFIGS: Cluster, and
  • CREATE: Topic
API key or OAuth
Cluster link Mirror topics Source cluster
  • DeveloperRead on Topics, and
  • DeveloperManage on Topics, or
  • CloudClusterAdmin (not recommended)
or
  • READ: Topics, and
  • DESCRIBE_CONFIGS: Topics
API key
Cluster link Sync consumer offsets Source cluster
  • DeveloperRead on Consumer Groups, or
  • Operator on Cluster, or
  • CloudClusterAdmin (not recommended)
or
  • DESCRIBE: Topics, and
  • READ: ConsumerGroups, and
  • DESCRIBE: ConsumerGroups
API key
Cluster link Sync ACLs Source cluster
  • Operator on Cluster, or
  • Operator on Cluster, or
  • CloudClusterAdmin (not recommended)
or DESCRIBE: Cluster API key
Cluster link Source-initiated cluster link to a Confluent Cloud cluster Destination cluster CloudClusterAdmin or ALTER: Cluster API key

Security Model

The Cluster Linking security model separates the permissions and credentials needed for each step of its lifecycle. This paradigm enables many teams to use cluster link operations on the topics that they already own. It also allows the owner of one cluster to share data over a cluster link with another team or organization, while only giving that team or organization READ access to the shareable, mirror topics.

Advantages

The advantages of this model are:

  • The access control lists (ACLs) needed to create a cluster link and mirror topics can be managed by the owner of the destination cluster.
  • Operations performed on a cluster link and mirror topics are available in Confluent Cloud Audit Logs.
  • You can use role-based access control (RBAC) to assign roles giving users or applications permissions on the destination cluster. These roles are described in Using role-based access control (RBAC) with Cluster Linking and Use Role-Based Access Control (RBAC) in Confluent Cloud.
  • The cluster link has its own credentials and ACLs on the source cluster. You can manage its READ access to the source cluster just like any other Apache Kafka® consumer. So, you can use your existing security workflows with cluster links.
    • If the source cluster is a Confluent Cloud cluster, it’s best to create a service account for the cluster link. You will issue source cluster credentials (API key and secret) to the cluster link’s service account.
    • If your source cluster is a Confluent Platform or open-source Apache Kafka cluster, the cluster link can use SASL/PLAIN or SASL/SCRAM. It cannot use Kerberos or OAuth.
  • The cluster link itself requires only read ACLs on the source cluster. No ACLs or credentials are required on the destination cluster in Confluent Cloud. This simplifies management.
  • When sharing data between teams, the owner of the source cluster can limit READ access to the mirror topics on the destination cluster on a topic-by-topic basis.

When sending data between the source and destination clusters, Cluster Linking uses Transport Layer Security (TLS) to encrypt data in transit.

If configured as such, Cluster Linking can also sync all client ACLs, or a configurable subset, from the source cluster to the destination cluster.

Building Blocks

The building blocks of Cluster Linking are:

  • A cluster link, which connects a source cluster to a destination cluster
  • A mirror topic, which exists on the destination cluster, and receives a copy of all messages that go to its source topic
  • A source topic, which is a regular topic that exists on the source cluster and is the source of data for the mirror topic

Cluster Linking Lifecycle and Permissions

Each phase of the Cluster Linking lifecycle requires different permissions. The phases are:

  • A user or client application creates a cluster link.
  • A user or client application creates a mirror topic using the cluster link.
  • The cluster link reads topic data, consumer offsets, and ACLs from the source cluster.
  • The cluster link writes topic data, consumer offsets, and ACLs to the destination cluster.
  • A user or client application modifies, promotes, or deletes the mirror topic.
  • A user or client application modifies or deletes the cluster link.
../../_images/cloud-cluster-linking-security-overview.png

Creating or modifying a mirror topic

Tip

If you want to make sure that particular topics cannot be mirrored by the cluster link under any circumstances, then do not give authorization (through RBAC or ACLs) for the cluster link’s principal on those topics.

To create or modify a cluster link, a user must be authenticated and authorized by either a service account or a Confluent Cloud user account.

If the user is authenticated with a Confluent Cloud user account, they must have the ClusterAdmin, EnvAdmin, or OrgAdmin RBAC role over the destination cluster in order to be authorized to create or modify mirror topics.

If the user or client application is authenticated with a service account, then their service account needs these ACLs on the destination cluster:

  • An ACL to allow them to ALTER the destination cluster.
  • An ACL to allow them to ALTERCONFIGS on the destination cluster.
  • An ACL to allow them to CREATE the mirror topic. This ACL can be one that allows them to create all topics, or it can be scoped by topic name(s).

ACLs required to list and describe mirror topics

The list mirrors command lists the mirror topics on a cluster and/or a cluster link. You can call this command through the Confluent Cloud API for Cluster Linking with `REST`: `GET /kafka/v3/clusters/<cluster-id>/links/<link-name>/mirrors` or from the unified Confluent CLI with confluent kafka mirror list.

The describe mirror command gives information about a specific mirror topic. You can call this command through the Confluent Cloud API with `REST`: `GET /kafka/v3/clusters/<cluster-id>/links/<link-name>/mirrors/<mirror-topic>` or from the Confluent CLI with confluent kafka mirror describe <mirror-topic>.

To list and describe mirror topics, you need one of the following ACLs:

  • The DESCRIBE ACL on a Cluster resource, which provides access to both list and describe for all topics on the cluster.
  • The DESCRIBE ACL on specific Topic(s), which provides access to both list and describe for the specified topics only.

Writing to the destination cluster

On Confluent Cloud, cluster links don’t need any extra permissions to write data to the destination cluster. Because they exist on the destination cluster, they already have the appropriate permissions.

Transport Layer Security (TLS)

For a cluster link between two Confluent Cloud clusters, data in transit is encrypted through one-way TLS V1.2 (in the Java community, this is referred to as one-way SSL).

For a cluster link from a Confluent Platform or Apache Kafka cluster to a Confluent Cloud cluster, data-in-transit must use one-way TLS (in the Java community, this is referred to as one-way SSL). It will default to the latest version of TLS that both clusters have enabled. TLS 1.2 and 1.3 are enabled by default in Apache Kafka.

Older versions of SSL can be enabled, but this is discouraged due to known security vulnerabilities.

Mutual TLS (mTLS)

Cluster Linking can use mTLS (two-way verification) for some, but not all, data exchanges:

  • Confluent Cloud to Confluent Cloud does not use mTLS; it uses TLS and SASL as described above in Transport Layer Security (TLS).
  • Data coming into Confluent Cloud from OSS Kafka can be configured to use mTLS
  • Data coming into Confluent Cloud from Confluent Platform can be configured to use either mTLS or source-initiated cluster links with TLS+SASL. mTLS cannot be used with source-initiated cluster links.

When you use mTLS or certificates, you must configure the Confluent Cloud cluster to authenticate to the Confluent Platform or Apache Kafka® source cluster using a certificate, keystore, and truststore. These must be stored on the cluster link’s configuration properties in PEM format (a Privacy Enhanced Mail Base64 encoded format). Pass this configuration in when creating the cluster link in the Confluent Cloud Console, the REST API, or the Confluent CLI. To learn more about PEM files and possible configurations, see the Apache Kafka KIP that introduced PEM file support.

Use these configuration properties (not all may be required):

security.protocol=SSL
ssl.truststore.type=PEM
ssl.keystore.type=PEM
ssl.keystore.certificate.chain
ssl.keystore.key
ssl.key.password
ssl.truststore.certificates

If your PEM format has newlines or \n, replace them with a single space (” “). Here’s an example configuration:

security.protocol=SSL
ssl.keystore.type=PEM
ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX ... xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX xXxXxXxXxXxXxXx -----END CERTIFICATE-----
ssl.keystore.key=-----BEGIN ENCRYPTED PRIVATE KEY----- xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX ... xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX xXxXxXxXxXxXxXxX -----END ENCRYPTED PRIVATE KEY-----
ssl.key.password=<REDACTED>
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX ... xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX xXxXxXxXxXxXx -----END CERTIFICATE-----

Public Internet Exposure

Unless otherwise noted, Cluster Linking between two clusters on Confluent Cloud uses public IP addresses and public DNS. However, this does not necessarily mean that the cluster linked data goes over the public internet:

  • Traffic between two clusters that are both on Amazon Web Services (AWS) always stays on AWS and never goes to the public internet. Per the AWS documentation: “When using public IP addresses, all communication between instances and services hosted in AWS use AWS’s private network.”
  • Traffic between two clusters that are both on Microsoft Azure always stays on Azure and never goes to the public internet. Per the Azure documentation: “If the destination address is for one of Azure’s services, Azure routes the traffic directly to the service over Azure’s backbone network, rather than routing the traffic to the Internet. Traffic between Azure services doesn’t traverse the Internet.”
  • All traffic is encrypted in transit. See Transport Layer Security (TLS).
  • Cluster Linking between clusters on two different cloud service providers or with a location not on a cloud service provider may cause traffic to route through the public internet.

Note

Traffic between a non Confluent Cloud cluster and a Confluent Cloud cluster might not use public IP addresses and/or public DNS.

Practical Examples

To see practical examples of how you might model security for different Cluster Linking deployments, refer to these use case tutorials:

Audit Logs

Whenever a user or application tries to create or delete a cluster link or a mirror topic, authentication and authorization events are created in Confluent Cloud Audit Logs. This allows you to monitor Cluster Linking access and activity throughout Confluent Cloud.

Using role-based access control (RBAC) with Cluster Linking

Below is a quick demo of using Cluster Linking with Confluent Cloud RBAC, followed by a summary of the roles needed on different resources.

Demo

This short video shows how to set up a cluster link across clusters with RBAC enabled, and make topic data available across clusters with different owners.

In the example, the owner of a Webstore cluster gives the Logistics cluster owner role-based access to all topics prefixed as public, such as orders, clicks, and products data.

The Logistics owner can then mirror Webstore topics on the Logistics cluster as if they were native, and get visibility into that external, streaming data.

Confluent Cloud RBAC roles

To enable secure Cluster Linking on Confluent Cloud, you must either grant RBAC roles to principals, as described in this section, or grant specific ACLs as previously described.

At least one of the following RBAC roles is required to create a cluster link:

The Cluster Link service account requires one of the following sets of Confluent Cloud RBAC roles:

  • (Recommended) DeveloperRead and DeveloperManage on the source cluster Topic(s) to be mirrored
  • (Not recommended for production): Any of the roles shown above for creating a cluster link

Tip

The Cluster Link does not need any roles on the destination cluster; only the roles noted above for the source cluster.

ACLs and consumer offset sync

To sync ACLs, the cluster link must have the following role:

To sync consumer offsets, the cluster link must have one of the following roles: