Confluent Developer Marketplace Get started
  1. Home
  2. Platform
  3. Confluent Private Cloud
  4. Multi-Tenancy

Migrate Existing Clusters to Multi-Tenant LKCs

Use Cluster Linking to migrate topics, consumer group offsets, and access control lists (ACLs) from an existing Kafka cluster to Logical Kafka Clusters (LKCs) on a multi-tenant Physical Kafka Cluster (PKC).

Cluster Linking keeps your applications running during replication until you complete the final cutover.

An LKC is an isolated, tenant-scoped Kafka cluster with its own bootstrap URL, role bindings, and quotas. A PKC is the multi-tenant Kafka cluster that hosts one or more LKCs. Clients and cluster link filters always use the same resource names that the LKC exposes, with no LKC-specific prefix.

In-place topic migration to LKCs isn’t supported, because LKCs manage only topics that were created through them. Instead, you deploy a new PKC, create an LKC for each team, and replicate data from the existing source cluster into each LKC over a cluster link.

Example scenario

This guide assumes a single shared source cluster where teams rely on topic-naming prefixes for logical separation:

Team

Source topics

Consumer groups

HR

hr_payroll, hr_employee_events

hr_payroll_app

Finance

finance_invoices, finance_payments

finance_billing_app

The source topics don’t need a common prefix. A cluster link migrates the topics that you select, so if topicA and topicB belong to the same team, one link can migrate both. The example uses prefixes only to group each team’s topics.

After the migration, strict multi-tenant isolation replaces the naming conventions. Each team receives a dedicated LKC with its own bootstrap URL, role bindings, and resource quotas.

The HR and Finance topics on the existing source cluster each replicate over their own cluster link into a dedicated read-only mirror topic set on the HR LKC and Finance LKC of a new multi-tenant PKC.

Migration steps

The platform operator performs Step 1 once. Complete Steps 2 through 7 for each team. The procedure shows the HR steps in detail, and the Finance steps repeat them:

Migration phases. Step 1 runs once, and Steps 2 to 7 repeat for each team.

Step

Action

Performed by

1

Deploy the destination PKC

Platform operator

2

Onboard an LKC for each team

Platform operator

3

Prepare the source cluster and the LKC client

Source cluster administrator and TenantAdmin

4

Create a cluster link from each LKC

TenantAdmin

5

Verify the mirrored data

TenantAdmin

6

Cut over the clients

TenantAdmin and application owners

7

Clean up migration resources

TenantAdmin and source cluster administrator

Until you promote a team’s mirror topics in Step 6, the source cluster remains that team’s system of record, and the mirror topics on its LKC are read-only. Teams migrate independently, so you can pause or postpone any team’s cutover at any point before you promote its topics.

Prerequisites

Before you start the migration, make sure that you have the following:

  • Software versions: Confluent Platform 8.4 or later and Confluent for Kubernetes (CFK) 3.4 or later for the destination PKC that the platform operator deploys in Step 1, with Multi-Tenancy enabled.

  • Network connectivity: Network paths that allow all brokers in the destination PKC to reach all brokers in the source cluster. Cluster links are destination-initiated, so the PKC brokers open all outbound connections.

  • Workload baseline: A snapshot of each team’s workload, including topic and partition counts on the source cluster, peak ingress and egress byte rates, the largest message size, and total connection counts.

  • Source credentials: A principal and credential on the source cluster for the cluster link. The source cluster administrator grants its permissions in Step 3.

  • Administrative access: A designated TenantAdmin for each team. In Step 2, the platform operator grants this person the TenantAdmin role on the team’s LKC, or the OrganizationAdmin role on the PKC. The DeveloperManage role isn’t enough, because it can’t create cluster links or promote mirror topics.

Step 1: Deploy the destination PKC

The platform operator deploys a new multi-tenant PKC by using CFK. All teams share this PKC after the migration. For deployment instructions, see the Deploy a Multi-Tenant Physical Kafka Cluster page.

Note

Don’t enable Multi-Tenancy on an existing cluster that has active workloads. Setting spec.multiTenant.enabled on an active cluster doesn’t assign existing topics to LKCs.

Step 2: Onboard an LKC for each team

The platform operator creates a LogicalKafkaCluster custom resource (CR) on the new PKC for each team and assigns the TenantAdmin role to the person who administers that team’s LKC. In the example, the platform operator onboards one LKC for HR and one for Finance. For instructions, see the Onboard an LKC to a Multi-Tenant Physical Kafka Cluster page.

Apply the guidance in the following sections to each LKC.

Size quotas and limits for the imported workload

Use the workload snapshot to set quotas.ingressByteRate, quotas.egressByteRate, quotas.brokerRequestThreads, limits.maxPartitions, and limits.maxMessageSizeBytes.

  • Mirror topics are ordinary topics in the LKC, so their partitions count toward limits.maxPartitions.

  • Set limits.maxMessageSizeBytes to the largest message size on the source topics, so that clients can still produce those messages after cutover.

Retrieve the LKC’s bootstrap URL

Retrieve the bootstrap URL from the LogicalKafkaCluster CR status. CFK generates the LKC ID, so you can’t choose the endpoint. The endpoint has the format <lkc-id>-kafka.<domain>:<port>.

kubectl get logicalkafkacluster <lkc-name> -n <your-namespace> \
  -o jsonpath='{.status.bootstrapEndpoint}'

Create role bindings before cutover

Configure LKC-scoped role-based access control (RBAC) role bindings for developer accounts before cutover. Although Cluster Linking syncs ACLs, it doesn’t sync RBAC role bindings. For instructions, see the Onboard a Developer page.

Step 3: Prepare the source cluster and the LKC client

The source cluster administrator and the TenantAdmin complete this step.

  1. Grant source permissions: On the source cluster, grant the following permissions to the link’s authentication principal (a user or service account). Without READ access to source topics, mirror topic creation fails.

    • DESCRIBE and READ on the source topics to migrate.

    • DESCRIBE on the consumer groups whose offsets you sync.

    • DESCRIBE on the cluster, if you sync ACLs.

    For the complete list, see Authorization (ACLs).

  2. Configure client properties: Create a client properties file that uses the TenantAdmin credentials from Step 2 to authenticate to the LKC’s bootstrap URL. LKCs require the SSL or SASL_SSL security protocol. This procedure refers to this file as <lkc-command-config>. For more information, see Authentication in Confluent Platform.

  3. Get the source cluster ID: Run the following command against the source cluster. The same ID applies to every team’s link, so retrieve it once and use it when creating each link:

    kafka-cluster cluster-id --bootstrap-server <source-bootstrap-servers> \
  --config <source-command-config>

Step 5: Verify the mirrored data

The TenantAdmin verifies each LKC independently by connecting directly to its bootstrap URL.

If you created the link with CFK, check the link and its mirror topics first:

kubectl get clusterlink hr-migration-link -n <pkc-namespace>

kubectl get clusterlink hr-migration-link -n <pkc-namespace> \
  -o jsonpath='{.status.mirrorTopics}'

The ClusterLink status shows the link’s state and the state of each mirror topic, but not the replication lag. Complete the following steps with the command-line tools to verify the lag, consumer offsets, and ACLs.

  1. Verify link configuration: Confirm the link status and active properties:

    kafka-cluster-links --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config> --list

kafka-configs --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config> --describe \
  --cluster-link hr-migration-link

    If a setting is missing, the LKC ignored it. For troubleshooting steps, see Troubleshooting.

  2. Verify topic replication: Verify that all filtered source topics exist as mirror topics in the ACTIVE state with minimal or zero replication lag:

    kafka-mirrors --describe --links hr-migration-link \
  --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config>

    Automatic mirror topic creation runs periodically, so mirror topics can take about five minutes to appear after you create the link. Mirror topics keep their original source names without LKC prefixes, for example, hr_payroll. Verify that other teams’ topics don’t appear in the LKC.

  3. Verify consumer offsets: Compare the committed offsets between the source cluster and the LKC:

    kafka-consumer-groups --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config> --describe \
  --group <hr-consumer-group>

    Offsets sync only while a topic is an active mirror and the consumer group has no active members on the LKC. Don’t start LKC consumers until offset verification completes.

  4. Verify ACL synchronization: Confirm that the synchronized ACLs exist:

    kafka-acls --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config> --list

    The LKC scopes each synced ACL to itself. If an ACL’s principal doesn’t exist in your identity provider, the ACL has no effect. Synced ACLs also count toward the per-LKC ACL limit that the brokers enforce (confluent.max.acls.per.tenant, default: 1000).

  5. Monitor system metrics: Observe PKC performance and resource usage during catch-up replication. For more information, see the Monitor and Manage Per-LKC Resource Quotas page.

Step 6: Cut over the clients

TenantAdmin users and application owners cut over one team at a time. Until you promote a team’s mirror topics, the source cluster stays the system of record for that team, so you can postpone cutover for any team without affecting the others.

This flow differs from the general migration tutorial, which moves consumer groups one at a time. In an LKC, offsets can’t be committed to a consumer group that has active members, so this procedure stops all consumers before promotion.

Caution

Cut over a team’s topics, producers, and consumers together in a single maintenance window. Promotion is irreversible. After you promote a topic, never restart source producers against the source topic, because the data streams permanently diverge.

Cutover sequence: stop source producers, stop source consumers and verify offsets, promote topics (irreversible), and then restart consumers and producers on the LKC, consumers first.

  1. Stop source producers: Stop all producer applications that publish to the team’s source topics. Messages that producers send to a source topic after you promote its mirror topic aren’t replicated to the LKC.

  2. Stop source consumers: Gracefully shut down the consumer applications on the source cluster. Wait for at least consumer.offset.sync.ms (default: 30 seconds), and then verify that the committed offsets on the LKC match the source cluster.

  3. Disable automatic topic creation: When the mirror lag of every topic reaches 0, disable automatic mirror topic creation so that the link doesn’t mirror new source topics:

    Set spec.mirrorTopicOptions.autoCreateTopics.enabled to false in hr-migration-link.yaml, and then apply the file again.

    kafka-configs --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config> --alter \
  --cluster-link hr-migration-link \
  --add-config auto.create.mirror.topics.enable=false

  4. Promote mirror topics: Convert the mirror topics into standard, writable topics:

    Add each topic to spec.mirrorTopics in hr-migration-link.yaml with the PROMOTE state, and then apply the file again:

    spec:
  mirrorTopics:
    - name: hr_payroll
      state: PROMOTE
    - name: hr_employee_events
      state: PROMOTE

    CFK promotes each topic and creates a KafkaTopic CR for it in the ClusterLink CR’s namespace. CFK derives the name of that CR from the topic name, so it can’t create the CR for a topic whose name contains a character that Kubernetes doesn’t allow in resource names, such as an underscore. CFK still promotes the topic.

    kafka-mirrors --promote \
  --topics <topic-1>,<topic-2> \
  --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config>

    Promotion verifies that there’s no lag, performs a final offset and configuration sync, and converts the mirror topics to standard topics. Promotion requires ALTER permission on the topics. If a consumer group has active members on the LKC, the final offset sync can’t complete, and the mirror topic stays in the PENDING_STOPPED state. If promotion doesn’t complete, see Troubleshooting. For failover options, see Stopping mirroring on a topic.

  5. Verify topic state: Confirm that all promoted topics reach the STOPPED state. Topics pass through the PENDING_STOPPED state first. To check the state, run the following command:

    kubectl get clusterlink hr-migration-link -n <pkc-namespace> \
  -o jsonpath='{.status.mirrorTopics}'

    kafka-mirrors --describe --include-stopped \
  --links hr-migration-link \
  --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config>

  6. Repoint client applications: Update the client application configurations to target the LKC’s status.bootstrapEndpoint value that you retrieved in Step 2. Check that clients use the SSL or SASL_SSL security protocol and trust the endpoint’s certificate. Confirm that the role bindings from Step 2 are in place. For more information, see the Onboard a Developer page. Start the consumer applications first, and then the producers.

    Warning

    Starting consumers on an LKC before offset sync completes can cause offset resets and duplicate message processing.

  7. Validate client operation: Verify that consumers resume processing from their committed offsets and that producers write successfully to the LKC’s topics.

Retain the original source topics until the migration is validated. The source cluster administrator deletes them in Step 7.

Step 7: Clean up migration resources

After client workloads run successfully on the LKC, the TenantAdmin and the source cluster administrator decommission the migration resources.

  1. TenantAdmin: Confirm that every mirror topic on the link is still STOPPED, as verified in Step 6. A link can’t be deleted while any of its mirror topics is in another state.

  2. TenantAdmin: Delete the cluster link:

    Delete the ClusterLink CR:

    kubectl delete clusterlink hr-migration-link -n <pkc-namespace>

    Before CFK deletes the link, it fails over any mirror topic that is still ACTIVE or PAUSED, and failover doesn’t wait for the lag to reach 0. Confirm that every mirror topic is STOPPED first.

    Keep the KafkaRestClass that you created for the LKC. The KafkaTopic CRs that CFK created for the promoted topics use it.

    kafka-cluster-links --bootstrap-server <lkc-bootstrap-endpoint> \
  --command-config <lkc-command-config> --delete \
  --link hr-migration-link

    Delete the link before you decommission the LKC or revoke the TenantAdmin credentials. For more information, see the Decommission an LKC page.

  3. Source cluster administrator: Revoke the link’s credential and remove the ACLs granted in Step 3.

  4. Source cluster administrator: Delete the migrated source topics, such as the hr_ topics.

After all teams migrate, retire the source cluster.

Troubleshooting

A source topic didn’t mirror

Check the following:

  • Filter mismatch: Verify that the topic names match the filter criteria.

  • Insufficient authorization: Confirm that the link principal holds READ permission on the source topic.

  • Naming collision: Confirm that a topic with the same name doesn’t already exist on the destination LKC.

  • Partition limits: Verify that the new mirror topics’ partitions don’t exceed limits.maxPartitions. If necessary, update the LogicalKafkaCluster CR. CFK sends the complete quotas and limits to Metadata Service (MDS) on every reconcile, so any quota or limit that you omit from the CR is cleared. Keep all existing quota and limit values in the CR when you change one.

Consumer offsets don’t match the source cluster

  • Wait at least consumer.offset.sync.ms after you stop the source consumers.

  • Check that the consumer group filter includes the consumer group. This is consumerGroupFilters in the ClusterLink CR, or consumer.offset.group.filters with the command-line tools.

  • Confirm that the consumer group has no active members on the LKC.

Promotion doesn’t complete

  • If lag exists, wait for the lag to reach 0, and then run the command again.

  • If a consumer group has active members on the LKC, stop them. The link can’t commit offsets for an active group, so the mirror topic stays in the PENDING_STOPPED state until the final offset sync succeeds. To see the error, run kafka-mirrors --describe against the LKC.

  • If the PKC’s brokers can’t reach the source cluster, restore connectivity. If the source cluster is down, run kafka-mirrors --failover instead. You lose any data that the link hasn’t mirrored yet. For more information, see Stopping mirroring on a topic.

Related content