# Migrate Topics Incrementally with Topic-Based Routing and Fencing Using dynamic routing capabilities, a single Confluent Gateway route can route topics to different Apache Kafka® clusters at the same time, so you can migrate topics without updating client bootstrap addresses or restarting applications. These clusters can be any Kafka cluster, including Confluent Platform, Confluent Cloud, and open-source Kafka. You can migrate topics to a new cluster individually or in batches. This guide explains how to use topic-based fencing and routing to migrate a batch of topics without disrupting clients. Before you begin, review the [limitations](#gateway-topic-based-routing-limitations). #### IMPORTANT Dynamic routing, which topic-based routing uses, is an Early Access Program feature in Confluent Private Cloud Gateway 1.4. Fencing is generally available. An Early Access feature is a component of Confluent Private Cloud Gateway introduced to gain feedback. This feature should be used only for evaluation and non-production testing purposes or to provide feedback to Confluent, particularly as it becomes more widely available in follow-on preview editions. Early Access Program features are intended for evaluation use in development and testing environments only, and not for production use. Early Access Program features are provided: (a) without support; (b) “AS IS”; and (c) without indemnification, warranty, or condition of any kind. No service level commitment will apply to Early Access Program features. Early Access Program features are considered to be a Proof of Concept as defined in the Confluent Private Cloud Gateway Terms of Service. Confluent may discontinue providing preview releases of the Early Access Program features at any time in Confluent’s sole discretion. Configuration schemas for dynamic routing, including property names and YAML structure, are subject to change before General Availability (GA) and might not be backward compatible. ## What topic-based routing and fencing do A static Confluent Gateway route sends all traffic to a single Kafka cluster. A dynamic route that defines topic-based rules can serve topics from more than one cluster. Fencing pauses traffic for selected topics at the topic level, in a way that clients retry, while you use Cluster Linking or Replicator to move the topics. Use topic-based routing to: - **Migrate gradually.** You move topics to the new cluster in batches, instead of cutting over all the topics at once. - **Keep clients unchanged.** Clients connect to one bootstrap address, and Confluent Gateway sends each request to the cluster that owns the topic. - **Avoid writes to both clusters.** Fencing pauses writes to a topic while its data finishes replicating, and producers retry the fenced requests. Fencing can also pause reads if you need it to. Topic-based routing differs from [Client Switchover](gateway-migrate.md#gateway-client-switchover), which moves an entire route to another cluster at once. ## How topic-based routing and fencing work Confluent Gateway models each Kafka cluster as a **streaming domain** ([Streaming domains configuration](gateway-deploy.md#gateway-config-streaming-domains-docker)). Topic-based routing and fencing build on this model and use three main components: | Component | Description | Configuration reference | |---------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Dynamic route | A route that uses topic-based routing. It binds more than one streaming
domain using `streamingDomains` to serve topics from more than one
cluster through one route. | [Routes configuration](gateway-deploy.md#gateway-config-routes-docker) | | Routing rules (`rules.routing`) | Maps topics to a streaming domain by exact name or pattern.
Confluent Gateway checks the rules in order, and the first match wins. Unmatched
topics fall back to the `default` streaming domain. | [Routing rules configuration (Early Access)](gateway-deploy.md#gateway-config-dynamic-routing-rules-docker) | | Fencing rules (`rules.fencing`) | Pauses or allows traffic for selected topics. Confluent Gateway checks
the rules in order, and the first match wins. Blocked requests return errors that
clients can retry, or empty responses, so most clients retry instead of
failing.
Fencing rules also work on static routes. | [Fencing rules configuration](gateway-deploy.md#gateway-config-fence-rules-docker)

[Responses to blocked requests](gateway-deploy.md#gateway-config-fence-rules-handling-docker) | ### Migration sequence Because only one cluster owns a topic at a time, migrating a topic requires the **replicate-fence-promote-reroute** workflow: 1. **Replicate:** Set up replication to the destination cluster. 2. **Fence:** Pause writes to the topics that you are about to migrate. 3. **Promote:** Promote the mirror topic on the destination cluster. Make sure that replication has caught up first. 4. **Reroute:** Update the routing rules to point to the new cluster, and remove the fence. For the step-by-step procedure, see [Migrate topics between clusters](#gateway-topic-based-routing-migrate). ## Limitations Before you plan a migration, review the following limitations of topic-based routing and fencing. A dynamic route has the following limitations: - A dynamic route supports only the `host` broker identification strategy. The `host` strategy requires `tls` configuration. - A dynamic route doesn’t support transactions or queues for Kafka (share groups). Confluent Gateway advertises these APIs as unsupported. It rejects requests from transactional producers with an `INVALID_REQUEST` error, which is fatal to the client, so transactional producers fail instead of retrying. - Before you convert a static route to a dynamic route, make sure that no transactional clients use the route, including Kafka Streams applications and Kafka Connect workers in exactly-once mode. Transactional producers that are running on the route when you convert it fail in the middle of a transaction. The broker aborts the open transaction only after `transaction.timeout.ms` expires, and `read_committed` consumers of the affected partitions can’t read past it until then. - Confluent Gateway doesn’t support the following APIs on a dynamic route. Unsupported requests return an `UNSUPPORTED_VERSION` error. - Consumer group management APIs, such as `DescribeGroups`, `ListGroups`, `DeleteGroups`, and `OffsetDelete`. As a result, the `kafka-consumer-groups` tool doesn’t work on a dynamic route. - Admin management APIs, such as the access control list (ACL) APIs (`CreateAcls` and `DeleteAcls`), `DescribeConfigs`, `AlterConfigs`, `CreatePartitions`, and `DeleteRecords`. - A dynamic route doesn’t support the `consumer` group protocol. A consumer with `group.protocol=consumer` fails at startup with a fatal exception, because Confluent Gateway doesn’t advertise the `ConsumerGroupHeartbeat` API. Set `group.protocol=classic` explicitly on your consumers, because Apache Kafka® plans to change the default in 5.0. A group that already uses the consumer protocol converts back to the classic protocol when it’s empty, so switch all of its members at once. - A dynamic route doesn’t support topic auto-creation. Clients must create topics with the `CreateTopics` API. - The clusters that a dynamic route binds can run different Kafka versions. For example, one cluster can run 3.9 and another can run 4.0. - When all authentication legs of a route use passthrough, Confluent Gateway sends the client’s Simple Authentication and Security Layer (SASL) credential to every bound cluster, so every cluster must accept the same credential. - Salted Challenge Response Authentication Mechanism (SCRAM) client authentication requires that you configure authentication swap for every cluster. Confluent Gateway doesn’t support SCRAM with passthrough, or for authenticating to a cluster. Use `PLAIN` or `OAUTHBEARER` instead. To configure authentication on a route, see [Route authentication configuration](gateway-security.md#gateway-authn-dynamic-docker). Fencing has the following limitations: - Fencing lets you pause only `PRODUCE` or `CONSUME` traffic for selected topics. For details, see [fencing rules](gateway-deploy.md#gateway-config-fence-rules-docker). Confluent Gateway doesn’t fence other operations, such as fetching metadata or creating and deleting topics. - Confluent Gateway doesn’t fence offset fetches. - A producer that uses `acks=0` can’t receive an error response, so Confluent Gateway silently drops the records for fenced topics. Use `acks=1` or `acks=all` for the topics that you migrate. ## Migrate topics between clusters Migrate topics between Kafka clusters in a Docker-based Confluent Gateway deployment without client downtime or application restarts. ### Example setup The migration sequence uses the following example environment: * **Topic:** `orders` * **Source cluster:** `cp` streaming domain (Confluent Platform) * **Destination cluster:** `cc` streaming domain (Confluent Cloud) #### NOTE This sequence applies to any set of topics and to any pair of clusters that Cluster Linking supports. The destination cluster must be able to host mirror topics. ### Prerequisites Before you begin, make sure that you have: - Confluent Gateway 1.4, deployed with Docker. For more information, see [Configure and Deploy Confluent Private Cloud Gateway using Docker](gateway-deploy.md#gateway-deploy-docker). - A streaming domain defined for each cluster. For more information, see [Streaming domains configuration](gateway-deploy.md#gateway-config-streaming-domains-docker). - A cluster link and mirror topics on the destination cluster, created outside Confluent Gateway. The destination cluster needs a mirror topic for every topic that the consumer groups of the route read. Create them before [step 1](#gateway-topic-based-routing-offsets-step), or enable auto-create mirror topics on the cluster link. Step 2 moves consumer group coordination to the destination cluster. The destination cluster rejects offset commits for topics that it doesn’t have. For more information, see [Mirror topics on Confluent Platform](/platform/current/multi-dc-deployments/cluster-linking/mirror-topics-cp.html) or [Mirror topics on Confluent Cloud](/cloud/current/multi-cloud/cluster-linking/mirror-topics-cc.html). - Client applications that connect to Confluent Gateway through a route. If your clients already connect through a static route, you convert that route in [step 2](#gateway-topic-based-routing-configure-step). A dynamic route supports only the `host` broker identification strategy, which requires an SSL keystore in the route’s `security` settings. If your existing route uses the `port` strategy, clients must change to the new broker addresses first. - All members of each consumer group connect to Confluent Gateway through the route at the same time. If some members still connect to the source cluster directly while `coordination.group` is the destination, the group is split across two coordinators and the members process partitions twice. - A fence can’t pause clients that connect to the source cluster directly. For those clients, stop all members of each consumer group, sync the offsets, and then start the members on the route. - Producers whose `delivery.timeout.ms` is longer than the time that you expect a topic to stay fenced. For more information, see [step 4](#gateway-topic-based-routing-fence-step). ### Prepare authentication With authentication swap, clients authenticate to Confluent Gateway with one set of credentials, and Confluent Gateway replaces them with its own credentials when it connects to each cluster. Configure `clientAuth` once for the clients, and `clusterAuth` for each streaming domain. A route that uses SCRAM client authentication requires authentication swap for every cluster. 1. Choose and configure a secret store that holds the credentials that Confluent Gateway uses to connect to each cluster. Use the secret store provider that fits your environment, such as AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, or CyberArk Conjur. Add the store to `gateway.secretStores` and note its `name`, because the `secretStore` setting in the `security` section that you add later in this procedure must use it. Store one secret for each client user. Confluent Gateway looks up the secret by the client’s username, and uses the username and password in it to connect to the cluster. Providers differ in how they locate and format a secret, for example in `prefixPath`, `separator`, and `useJson`. For the settings and the secret format of your provider, see [Secret store configuration](gateway-security.md#gateway-secret-stores-docker). 2. Create the Java Authentication and Authorization Service (JAAS) file for client authentication, for example `/etc/gateway/client-plain-jaas.conf`. It lists the users that connect to Confluent Gateway, with one `user_` property for each user. For example: ```properties org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="admin-secret" user_additional-user="additionaluser-secret"; ``` For more information, see [Client authentication for authentication swapping](gateway-security.md#gateway-authn-swap-client-docker). 3. Create a JAAS template file for each cluster, for example `/etc/gateway/cp-swap-jaas.conf` and `/etc/gateway/cc-swap-jaas.conf`. Keep the `%s` placeholders as they are. Confluent Gateway fills them with the username and password that it reads from the secret store. ```properties org.apache.kafka.common.security.plain.PlainLoginModule required username="%s" password="%s"; ``` For more information, see [Cluster authentication for authentication swapping](gateway-security.md#gateway-authn-swap-cluster-docker). 4. Mount the three JAAS files, and any other file that your secret store configuration references, into the Confluent Gateway container at the paths that you set in the configuration, for example with a `volumes` entry in `docker-compose.yaml`. For the settings of your secret store, see [Secret store configuration](gateway-security.md#gateway-secret-stores-docker). 5. Add the `security` section to the route. The following example shows only the `security` section of the route: ```yaml gateway: routes: - name: migration-route security: clientAuth: sasl: mechanism: PLAIN callbackHandlerClass: org.apache.kafka.common.security.plain.internals.PlainServerCallbackHandler jaasConfig: file: /etc/gateway/client-plain-jaas.conf clusterAuth: cp: auth: swap secretStore: kms-store sasl: mechanism: PLAIN callbackHandlerClass: org.apache.kafka.common.security.authenticator.SaslClientCallbackHandler jaasConfig: file: /etc/gateway/cp-swap-jaas.conf cc: auth: swap secretStore: kms-store sasl: mechanism: PLAIN callbackHandlerClass: org.apache.kafka.common.security.authenticator.SaslClientCallbackHandler jaasConfig: file: /etc/gateway/cc-swap-jaas.conf ``` The example uses `kms-store` as the secret store name. Replace it with the `name` of the secret store that you configured earlier in this section. Each key under `clusterAuth` must match the name of a streaming domain that the route binds, which in this example is `cp` or `cc`. Keep the `security` section ready. You add it to the route in [step 2](#gateway-topic-based-routing-configure-step), so that the authentication settings take effect in the same configuration change as the dynamic route. For all authentication settings, see [Route authentication configuration](gateway-security.md#gateway-authn-dynamic-docker). ### Step 1: Fence consumers and sync offsets to the destination Step 2 moves consumer group coordination, including the committed offsets, to the destination cluster. The offsets must be on the destination cluster before that happens, and they must not change while you copy them. To stop them from changing, fence the entire consume plane first. #### NOTE To defer the offset sync, set `coordination.group` to the source cluster in [step 2](#gateway-topic-based-routing-configure-step), and skip this step. Sync the offsets to the destination after you migrate all topics, as [step 8](#gateway-topic-based-routing-complete-step) describes. You apply every configuration change in this guide the same way. How you apply a change depends on how you provide the configuration: * If the configuration is inline in `docker-compose.yaml`, as in the `GATEWAY_CONFIG` environment variable, Confluent Gateway reads it only at startup. Recreate the container to apply a change: ```bash docker compose -f docker-compose.yaml up -d ``` * If you mount the configuration file and set `GATEWAY_CONFIG_FILE`, Confluent Gateway applies route and `rules` changes without a restart. This requires a license other than the Trial license. With a Trial license, restart Confluent Gateway to apply the changes. Applying a change to a route replaces the route and drops all of its client connections. The clients reconnect. When Confluent Gateway applies a change without a restart, it rejects an invalid change and keeps the previous configuration, so check the Confluent Gateway log after you apply a change. If you restart Confluent Gateway, an invalid configuration stops it from starting. 1. Block all consume traffic on the route that your clients use. Add the following fencing rule, which has no `topics`, and apply the configuration. If your clients already connect through a static route, add the rule to that route. ```yaml gateway: routes: - name: migration-route rules: fencing: - trafficType: CONSUME blocked: true ``` A rule without `topics` also blocks the `JoinGroup`, `SyncGroup`, and `Heartbeat` requests, which don’t name a topic. A rule that lists topics doesn’t block them. Consumers can’t change offsets while the fence is in place. Producers aren’t affected. If the route uses the deprecated `fence` setting, replace it with `rules.fencing`, because a route can’t use both. 2. Sync the consumer group offsets from the source cluster to the destination cluster, for example with consumer offset sync on the cluster link. Confirm that the offsets on the destination match the source. 3. Disable consumer offset sync on the cluster link, and on any other replicator, so that it can’t overwrite the offsets that clients commit through Confluent Gateway afterward. ### Step 2: Configure a dynamic route and remove the consumer fence Make the following changes in a single configuration change: switch to the dynamic route with `coordination.group` set to the destination, add the `security` section that you prepared, and remove the consumer fence rule from step 1. Consumers then reconnect and find their coordinator on the destination cluster. Bind both streaming domains to the route. Set the source cluster as the `default` owner of all topics. Clients continue to reach every topic through the same bootstrap address. If your clients already connect through a static route, update that route instead of adding a new one. Use `rules.fencing` instead of the deprecated `fence` setting, because a route can’t use both. For more information, see [Fencing filter](gateway-deploy.md#gateway-config-fencing-docker). ```yaml gateway: routes: - name: migration-route endpoint: migration-gw.example.com:9092 mode: DYNAMIC brokerIdentificationStrategy: type: host pattern: broker-$(nodeId).migration-gw.example.com:9092 streamingDomains: - name: cp bootstrapServerId: cp-bootstrap - name: cc bootstrapServerId: cc-bootstrap rules: routing: coordination: group: cc default: cp ``` This example shows only the settings that topic-based routing requires. A route that uses `type: host` also requires an SSL keystore in its `security` settings. For more information, see [Configure Security for Confluent Private Cloud Gateway](gateway-security.md#gateway-security-docker). Add the `security` section that you prepared to the same route, and then apply the configuration the same way that you did in [step 1](#gateway-topic-based-routing-apply-step). The `coordination.group` setting chooses the streaming domain that coordinates all consumer groups on the route, including their committed offsets. This example uses the destination, `cc`. Every topic that the groups read must exist on that cluster, because the cluster rejects offset commits for topics that it doesn’t have. For all routing settings, see [Routing rules configuration (Early Access)](gateway-deploy.md#gateway-config-dynamic-routing-rules-docker). ### Step 3: Check the mirror topic on the destination cluster Confirm that the mirror topic for `orders` exists on the destination cluster and is replicating. If it doesn’t exist, use the cluster link to create it. Confluent Gateway continues to send all `orders` traffic to the source cluster while the data replicates. ### Step 4: Fence produce requests to the topic Add a fencing rule that blocks produce requests to `orders`. Fencing produce requests stops new writes so that replication can catch up. Consumers can keep reading the data that already exists. Producers receive an error that they can retry, and they keep retrying until you remove the fence or their `delivery.timeout.ms` expires. The default is 120,000 ms, which is two minutes. When the timeout expires, the producer stops retrying and returns an error to the application. To keep the fence short, remove it as soon as you update the routing in [step 6](#gateway-topic-based-routing-reroute-step). If you expect the fence to last longer than the timeout, increase `delivery.timeout.ms` for your producers before you fence the topic. Add the `fencing` list to the `rules` section of the route, as the following example shows. The other route settings don’t change. If you already moved other topics, keep their conditions in `rules.routing`. ```yaml gateway: routes: - name: migration-route rules: routing: coordination: group: cc default: cp fencing: - trafficType: PRODUCE topics: ["orders"] blocked: true ``` Apply the configuration to Confluent Gateway the same way that you did in [step 1](#gateway-topic-based-routing-apply-step). For more information about fencing rules and the errors that clients receive, see [Fencing rules configuration](gateway-deploy.md#gateway-config-fence-rules-docker). ### Step 5: Promote the mirror topic After the mirror topic catches up with the source topic, promote the mirror topic on the destination cluster. For information about how to check the replication lag and promote a mirror topic, see [Mirror topics on Confluent Platform](/platform/current/multi-dc-deployments/cluster-linking/mirror-topics-cp.html) or [Mirror topics on Confluent Cloud](/cloud/current/multi-cloud/cluster-linking/mirror-topics-cc.html). ### Step 6: Update the routing and remove the fence Add a condition that routes `orders` to the destination cluster. Remove the `fencing` entry for `orders`, or the entire `rules.fencing` list if it has no other rules. Confluent Gateway evaluates conditions before it applies the `default` streaming domain. As a result, `orders` goes to `cc`, and all other topics stay on `cp`. The following example shows only the `rules` section of the route: ```yaml gateway: routes: - name: migration-route rules: routing: coordination: group: cc conditions: - topics: ["orders"] streamingDomain: cc default: cp ``` Apply the updated configuration to Confluent Gateway, as in [step 1](#gateway-topic-based-routing-apply-step). Producers resume writing, and Confluent Gateway sends all `orders` traffic to the destination cluster. #### WARNING Change the routing and remove the fence in a single configuration change. If you remove the fence first, producers write to the source cluster after you promoted the mirror topic. Confluent Gateway returns a `NOT_LEADER_OR_FOLLOWER` error, which clients can retry, to clients that still hold metadata that points to the source cluster. Instead of reading from or writing to the source topic, the clients refresh their metadata. ### Step 7: Verify the migration Confirm that producers and consumers of `orders` work without any changes to their configuration. Then check the producer and consumer bandwidth of the Kafka clusters for the `orders` topic. After the migration, the bandwidth decreases on the source cluster and increases on the destination cluster. For other metrics that you can monitor on a route, see [Monitor topic-based routing and fencing](gateway-metrics-monitoring.md#gateway-metrics-topic-based-routing) and [Monitor Confluent Private Cloud Gateway Metrics](gateway-metrics-monitoring.md#gateway-metrics-monitoring). ### Step 8: Complete the migration Repeat [step 3](#gateway-topic-based-routing-mirror-step) through [step 7](#gateway-topic-based-routing-verify-step) for each remaining topic or group of topics. Keep the conditions for the topics that you already moved. To migrate more than one topic at once, list the topics in `topics`, or match them with `topicPatterns`, in both the fencing rule in [step 4](#gateway-topic-based-routing-fence-step) and the routing condition in [step 6](#gateway-topic-based-routing-reroute-step). For more information, see [Fencing rules configuration](gateway-deploy.md#gateway-config-fence-rules-docker) and [Routing rules configuration (Early Access)](gateway-deploy.md#gateway-config-dynamic-routing-rules-docker). After all topics have moved, the route no longer needs topic-based routing. Convert it back to a static route on the destination cluster, and remove `rules.routing` entirely. Complete the following steps: 1. List the topics on the source cluster and confirm that each one has a routing condition that points to the destination streaming domain. Topics that clients created during the migration go to the `default` streaming domain, which is the source cluster. Migrate any topics that are left first. 2. If `coordination.group` points to the source cluster, move the consumer group coordinator first, in the same order as [step 1](#gateway-topic-based-routing-offsets-step): fence the consume plane, and copy the committed offsets of all consumer groups to the destination cluster. Copy the offsets yourself, and include the offsets of every promoted topic, because Cluster Linking doesn’t sync offsets for promoted topics. If `coordination.group` already points to the destination, skip this step, because the static route keeps the same coordinator. 3. In a single configuration change, convert the route to a static route: - Replace `streamingDomains` with `streamingDomain` set to the destination streaming domain. - Remove `mode: DYNAMIC` and the entire `rules.routing` section, including `conditions`, `default`, and `coordination`. - Remove any remaining `rules.fencing` entries, including the consume-plane fence from the previous step. - Optionally, keep `clusterAuth` only for the destination streaming domain. 4. Apply the configuration. Clients reconnect to the static route without changing their bootstrap address. #### WARNING Don’t retire the source cluster until you apply the configuration that unbinds it from the route. If a retired cluster is still bound, requests that Confluent Gateway sends to every bound cluster fail. ## Next steps - To monitor routes during a migration, see [Monitor Confluent Private Cloud Gateway Metrics](gateway-metrics-monitoring.md#gateway-metrics-monitoring). - To move an entire route at once, see [Migrate between Kafka Clusters using Confluent Private Cloud Gateway](gateway-migrate.md#gateway-client-switchover).