# Migrate Topics with Topic-Based Routing and Fencing using Confluent for Kubernetes This topic explains how to migrate topics between Apache Kafka® clusters incrementally by updating the Confluent Gateway custom resource (CR) in Confluent for Kubernetes (CFK). Clients keep the same bootstrap address and don’t restart. Topic-based routing lets a single Confluent Gateway route serve topics from more than one Kafka cluster, and fencing temporarily pauses traffic for selected topics while you move them. You configure both in the `routes` section of the Confluent Gateway CR and apply each change with `kubectl apply`. To learn how topic-based routing and fencing work, and how they compare with Client Switchover, see [Migrate Topics Incrementally with Topic-Based Routing and Fencing](https://docs.confluent.io/private-cloud-gateway/current/gateway-topic-based-routing.html). #### IMPORTANT Dynamic routing, which topic-based routing uses, is an Early Access Program feature in Confluent Gateway 1.4 and Confluent for Kubernetes 3.4. Fencing is generally available. Configuration schemas for dynamic routing, including property names and YAML structure, are subject to change before General Availability (GA) and might not be backward compatible. To review the limitations, see [Limitations](https://docs.confluent.io/private-cloud-gateway/current/gateway-topic-based-routing.html#limitations). ## Prerequisites Before you begin, make sure that you have: * Confluent Gateway 1.4 deployed with CFK 3.4. For more information, see [Configure and Deploy Confluent Gateway using Confluent for Kubernetes](co-gateway-deploy.md#co-gateway-deploy). * A streaming domain defined for each cluster. For more information, see [Configure streaming domains](co-gateway-deploy.md#co-gateway-config-streaming-domains). * A cluster link from the source cluster to the destination cluster, and mirror topics on the destination cluster for every topic that the consumer groups of the route read, created outside Confluent Gateway. Create the mirror topics before [step 1](#co-gateway-tbr-offsets-step), or enable auto-create mirror topics on the cluster link. Step 2 moves consumer group coordination to the destination cluster, which rejects offset commits for topics that it doesn’t have. * Client applications that connect to Confluent Gateway through a route. Check your existing route: * If your clients connect through a static route, you convert that route to a dynamic route in [step 2](#co-gateway-tbr-configure-step). * A dynamic route requires the `host` broker identification strategy and the `security.client.tls` setting. * If your route uses the `port` strategy, update your clients to use the new broker addresses before you start. * All members of each consumer group connect to Confluent Gateway through the route at the same time. If some members still connect to the source cluster directly while `coordination.group` is the destination, the group is split across two coordinators and the members process partitions twice. * A fence can’t pause clients that connect to the source cluster directly. For those clients, stop all members of each consumer group, sync the offsets, and then start the members on the route. * Producers configured with `acks=1` or `acks=all`. For more information, see [Limitations](https://docs.confluent.io/private-cloud-gateway/current/gateway-topic-based-routing.html#limitations). * Producers whose `delivery.timeout.ms` is longer than the time that you expect a topic to stay fenced. For more information, see [step 4](#co-gateway-tbr-fence-step). * To apply changes without rolling the Confluent Gateway pods, a license other than the Trial license. With the Trial license, CFK rolls the pods for each change. For more information, see [step 1](#co-gateway-tbr-offsets-step). ## Migration steps Complete the following steps to move the `orders` topic from the source cluster to the destination cluster. The migration sequence uses the following example environment: * **Topic:** `orders` * **Source cluster:** `cp` streaming domain (Confluent Platform) * **Destination cluster:** `cc` streaming domain (Confluent Cloud) This sequence applies to any set of topics and to any pair of clusters that Cluster Linking supports. The destination cluster must be able to host mirror topics. ### Step 1: Fence consumers and sync offsets to the destination Step 2 moves consumer group coordination, including the committed offsets, to the destination cluster. The offsets must be on the destination cluster before that happens, and they must not change while you copy them. To stop them from changing, fence the entire consume plane first. #### NOTE To defer the offset sync, set `coordination.group` to the source cluster in [step 2](#co-gateway-tbr-configure-step), and skip this step. Sync the offsets to the destination after you migrate all topics, as [step 8](#co-gateway-tbr-complete-step) describes. The `spec.hotReload` setting controls whether Confluent Gateway applies each change in this procedure without restarting its pods or CFK rolls the Confluent Gateway pods: * If `spec.hotReload.enabled` is `true` and you use a license other than the Trial license, Confluent Gateway applies changes to `routes`, including `rules`, without rolling its pods. By default, `spec.hotReload.enableCanary` is `true`, so Confluent Gateway validates each change in a canary pod before it applies the change to the other pods. ```yaml kind: Gateway spec: hotReload: enabled: true ``` * If `spec.hotReload.enabled` is `false`, which is the default, CFK rolls the Confluent Gateway pods to apply each change. * In Trial mode, Confluent Gateway can’t apply changes without a restart. If hot reload is enabled, CFK doesn’t roll the pods, so your changes don’t take effect until you restart them. Keep hot reload disabled and let CFK roll the pods. If you don’t use the Trial license, enable hot reload now, because every step in this procedure applies the CR. With hot reload enabled, fencing and rerouting don’t roll the pods. Applying a change to a route replaces the route and drops all of its client connections. The clients reconnect. 1. Block all consume traffic on the route that your clients use. Add the following fencing rule, which has no `topics`, to the `rules` section of the route in your CR file. If your clients already connect through a static route, add the rule to that route. ```yaml kind: Gateway spec: routes: - name: migration-route rules: fencing: - trafficType: CONSUME blocked: true ``` A rule without `topics` also blocks the `JoinGroup`, `SyncGroup`, and `Heartbeat` requests, which don’t name a topic. A rule that lists topics doesn’t block them. Consumers can’t change offsets while the fence is in place. Producers aren’t affected. If the route uses the deprecated `fence` setting, replace it with `rules.fencing`, because a route can’t use both. 2. Apply the CR: ```bash kubectl apply -f ``` 3. Wait until every Confluent Gateway pod has applied the change. With hot reload, each pod picks up the change at its own time, so check the log of each pod for the `Hot-reload successful` message. Without hot reload, wait for the pods to roll, for example with `kubectl rollout status`. Until every pod has the fence, a pod without it still lets consumers commit offsets on the source cluster, and the copy can miss those commits. 4. Sync the consumer group offsets from the source cluster to the destination cluster, for example with consumer offset sync on the cluster link. Confirm that the offsets on the destination match the source. 5. **Disable consumer offset sync on the cluster link**, and on any other replicator, so that it can’t overwrite the offsets that clients commit through Confluent Gateway afterward. ### Step 2: Configure a dynamic route and remove the consumer fence Make the following changes in a single update: switch to the dynamic route with `coordination.group` set to the destination, and remove the consumer fence rule from step 1. Consumers then reconnect and find their coordinator on the destination cluster. Bind the `cp` and `cc` streaming domains to the route. * Set `cp`, the source cluster, as the `default` owner of all topics. Clients continue to reach every topic through the same bootstrap address. * If your clients already connect through a static route, update that route instead of adding a new one. * Don’t set the `fence` setting on the route, because a route can’t use `fence` and `rules.fencing` together. For more information, see [Fencing filter](co-gateway-deploy.md#co-gateway-config-fencing). ```yaml kind: Gateway spec: routes: - name: migration-route endpoint: migration-gw.example.com:9092 mode: DYNAMIC brokerIdentificationStrategy: type: host pattern: broker-$(nodeId).migration-gw.example.com:9092 streamingDomains: - name: cp bootstrapServerId: cp-bootstrap - name: cc bootstrapServerId: cc-bootstrap security: client: tls: secretRef: gateway-tls rules: routing: coordination: group: cc default: cp ``` This example shows only the settings that topic-based routing requires. You must also define the `cp` and `cc` streaming domains in `spec.streamingDomains`. For more information, see [Configure streaming domains](co-gateway-deploy.md#co-gateway-config-streaming-domains). Apply the CR: ```bash kubectl apply -f ``` Set `coordination.group` to the destination, `cc`, so that Confluent Gateway coordinates consumer groups and their committed offsets on that cluster. Every topic that the groups read must exist there. For all routing settings, see [Configure routing rules (Early Access)](co-gateway-deploy.md#co-gateway-config-routing-rules). ### Step 3: Check the mirror topic on the destination cluster Confirm that the mirror topic for `orders` exists on the destination cluster and is replicating. The prerequisites require mirror topics for every topic that the consumer groups read. If the mirror topic doesn’t exist, use the cluster link to create it. For more information, see [Mirror topics on Confluent Platform](/platform/current/multi-dc-deployments/cluster-linking/mirror-topics-cp.html) or [Mirror topics on Confluent Cloud](/cloud/current/multi-cloud/cluster-linking/mirror-topics-cc.html). Confluent Gateway continues to send all `orders` traffic to the source cluster while the data replicates. ### Step 4: Fence produce requests to the topic Add a fencing rule that blocks produce requests to `orders`. Fencing produce requests stops new writes so that replication can catch up. Consumers can keep reading the data that already exists. Producers receive an error that they can retry, and they keep retrying until you remove the fence or their `delivery.timeout.ms` expires. The default is 120,000 ms, which is two minutes. When the timeout expires, the producer stops retrying and returns an error to the application. To keep the fence short, remove it as soon as you update the routing in [step 6](#co-gateway-tbr-reroute-step). If you expect the fence to last longer than the timeout, increase `delivery.timeout.ms` for your producers before you fence the topic. Add the `fencing` list to the `rules` section of the route in your CR file, as the following example shows. The other route settings don’t change. If you already moved other topics, keep their conditions in `rules.routing`. ```yaml kind: Gateway spec: routes: - name: migration-route rules: routing: coordination: group: cc default: cp fencing: - trafficType: PRODUCE topics: ["orders"] blocked: true ``` Apply the CR: ```bash kubectl apply -f ``` For more information about fencing rules and the errors that clients receive, see [Configure fencing rules](co-gateway-deploy.md#co-gateway-config-fencing-rules). ### Step 5: Promote the mirror topic Before you promote the mirror topic, confirm that every Confluent Gateway pod has applied the produce fence from [step 4](#co-gateway-tbr-fence-step), as described in [step 1](#co-gateway-tbr-offsets-step). A pod that doesn’t have the fence yet keeps writing to the source cluster after the promotion, and those writes never reach the destination cluster. After the mirror topic catches up with the source topic, promote the mirror topic on the destination cluster. For information about how to check the replication lag and promote a mirror topic, see [Mirror topics on Confluent Platform](/platform/current/multi-dc-deployments/cluster-linking/mirror-topics-cp.html) or [Mirror topics on Confluent Cloud](/cloud/current/multi-cloud/cluster-linking/mirror-topics-cc.html). ### Step 6: Update the routing and remove the fence In the `rules` section of the route in your CR file, make the following changes: * Add a condition to `rules.routing` that routes `orders` to the destination cluster, `cc`. * Remove the `fencing` entry for `orders`. If `rules.fencing` has no other rules, remove the entire list. Confluent Gateway evaluates conditions before it applies the `default` streaming domain, so `orders` goes to `cc` and all other topics stay on `cp`. The following example shows only the `rules` section of the route: ```yaml kind: Gateway spec: routes: - name: migration-route rules: routing: coordination: group: cc conditions: - topics: ["orders"] streamingDomain: cc default: cp ``` Apply the CR: ```bash kubectl apply -f ``` Producers resume writing, and Confluent Gateway sends all `orders` traffic to the destination cluster. #### WARNING Change the routing and remove the fence in a single CR change. If you remove the fence first, producers write to the source cluster after you promoted the mirror topic. For a short time after you apply the change, clients that still have metadata for the source cluster might receive `NOT_LEADER_OR_FOLLOWER` errors. This is expected, because Confluent Gateway rejects requests to the topic on the source cluster. Clients refresh their metadata and retry the request. ### Step 7: Verify the migration * On the destination cluster, the producer and consumer bandwidth for `orders` increases. * The rate of `gateway_fencing_v2_requests_total` for the `orders` topic is non-zero while the fence is active, and drops to zero after you remove the fence. * The rate of `gateway_router_rejections_total` with `reason="cross-cluster-ownership"` drops to zero after clients refresh their metadata. * `gateway_router_metadata_topics_dropped_total` with `reason="owned-elsewhere"` stops increasing. * `gateway_router_fanout_leg_failures_total` doesn’t increase. Confluent Gateway serves the metrics on the admin port `9190` by default. To read them from a Confluent Gateway pod, forward the port and request the metrics endpoint: ```bash kubectl port-forward pod/ 9190:9190 ``` ```bash curl -s http://localhost:9190/metrics | grep gateway_router ``` For more information about these metrics and other metrics to monitor, see [Monitor topic-based routing and fencing](https://docs.confluent.io/private-cloud-gateway/current/gateway-metrics-monitoring.html#monitor-topic-based-routing-and-fencing) in the Confluent Private Cloud Gateway metrics reference. ### Step 8: Complete the migration Repeat [step 3](#co-gateway-tbr-mirror-step) through [step 7](#co-gateway-tbr-verify-step) for each remaining topic or group of topics. Keep the conditions for the topics that you already moved. To migrate more than one topic at once, list the topics in `topics`, or match them with `topicPatterns`, in both the fencing rule in [step 4](#co-gateway-tbr-fence-step) and the routing condition in [step 6](#co-gateway-tbr-reroute-step). For more information, see [Configure fencing rules](co-gateway-deploy.md#co-gateway-config-fencing-rules) and [Configure routing rules (Early Access)](co-gateway-deploy.md#co-gateway-config-routing-rules). After all topics have moved, the route no longer needs topic-based routing. Convert it back to a static route on the destination cluster, and remove `rules.routing` entirely. Complete the following steps: 1. List the topics on the source cluster and confirm that each one has a routing condition that points to the destination streaming domain. Topics that clients created during the migration use the `default` streaming domain, which is the source cluster. Migrate any topics that are left first. 2. If `coordination.group` points to the source cluster, move the consumer group coordinator first, in the same order as [step 1](#co-gateway-tbr-offsets-step): fence the consume plane, wait until every Confluent Gateway pod has applied the fence, and copy the committed offsets of all consumer groups to the destination cluster. Copy the offsets yourself, and include the offsets of every promoted topic, because Cluster Linking doesn’t sync offsets for promoted topics. If `coordination.group` already points to the destination, skip this step, because the static route keeps the same coordinator. 3. In a single CR change, convert the route to a static route: - Replace `streamingDomains` with `streamingDomain` that references the destination streaming domain by `name` and `bootstrapServerId`. - Remove `mode: DYNAMIC` and the entire `rules.routing` section, including `conditions`, `default`, and `coordination`. - Remove any remaining `rules.fencing` entries, including the consume-plane fence from the previous step. - Keep the `security.cluster` settings only for the destination streaming domain, if you configured any. 4. Apply the CR: ```bash kubectl apply -f ``` Clients reconnect to the static route without changing their bootstrap address. #### WARNING Don’t retire the source cluster until you apply the CR that unbinds it from the route. If you retire a cluster before you unbind it, requests that Confluent Gateway sends to every bound cluster fail. ## Next steps * To diagnose problems during a migration, see [Troubleshoot Confluent Gateway CFK Deployments](co-gateway-troubleshoot.md#co-gateway-troubleshoot). * To move an entire route at once, see [Migrate between Kafka Clusters using Confluent Private Cloud Gateway](https://docs.confluent.io/private-cloud-gateway/current/gateway-migrate.html).