Confluent Developer Marketplace Get started
  1. Home
  2. Platform
  3. CFK
  4. Configure and Manage Confluent Private Cloud Gateway with Confluent for Kubernetes

Migrate Topics with Topic-Based Routing and Fencing using Confluent for Kubernetes

This topic explains how to migrate topics between Apache Kafka® clusters incrementally by updating the Confluent Gateway custom resource (CR) in Confluent for Kubernetes (CFK). Clients keep the same bootstrap address and don’t restart.

Topic-based routing lets a single Confluent Gateway route serve topics from more than one Kafka cluster, and fencing temporarily pauses traffic for selected topics while you move them. You configure both in the routes section of the Confluent Gateway CR and apply each change with kubectl apply.

To learn how topic-based routing and fencing work, and how they compare with Client Switchover, see Migrate Topics Incrementally with Topic-Based Routing and Fencing.

Important

Dynamic routing, which topic-based routing uses, is an Early Access Program feature in Confluent Gateway 1.4 and Confluent for Kubernetes 3.4. Fencing is generally available. Configuration schemas for dynamic routing, including property names and YAML structure, are subject to change before General Availability (GA) and might not be backward compatible.

To review the limitations, see Limitations.

Prerequisites

Before you begin, make sure that you have:

  • Confluent Gateway 1.4 deployed with CFK 3.4. For more information, see Configure and Deploy Confluent Gateway using Confluent for Kubernetes.

  • A streaming domain defined for each cluster. For more information, see Configure streaming domains.

  • A cluster link from the source cluster to the destination cluster, and mirror topics on the destination cluster for every topic that the consumer groups of the route read, created outside Confluent Gateway. Create the mirror topics before step 1, or enable auto-create mirror topics on the cluster link. Step 2 moves consumer group coordination to the destination cluster, which rejects offset commits for topics that it doesn’t have.

  • Client applications that connect to Confluent Gateway through a route. Check your existing route:

    • If your clients connect through a static route, you convert that route to a dynamic route in step 2.

    • A dynamic route requires the host broker identification strategy and the security.client.tls setting.

    • If your route uses the port strategy, update your clients to use the new broker addresses before you start.

  • All members of each consumer group connect to Confluent Gateway through the route at the same time. If some members still connect to the source cluster directly while coordination.group is the destination, the group is split across two coordinators and the members process partitions twice.

  • A fence can’t pause clients that connect to the source cluster directly. For those clients, stop all members of each consumer group, sync the offsets, and then start the members on the route.

  • Producers configured with acks=1 or acks=all. For more information, see Limitations.

  • Producers whose delivery.timeout.ms is longer than the time that you expect a topic to stay fenced. For more information, see step 4.

  • To apply changes without rolling the Confluent Gateway pods, a license other than the Trial license. With the Trial license, CFK rolls the pods for each change. For more information, see step 1.

Migration steps

Complete the following steps to move the orders topic from the source cluster to the destination cluster.

The migration sequence uses the following example environment:

  • Topic: orders

  • Source cluster: cp streaming domain (Confluent Platform)

  • Destination cluster: cc streaming domain (Confluent Cloud)

This sequence applies to any set of topics and to any pair of clusters that Cluster Linking supports. The destination cluster must be able to host mirror topics.

Step 1: Fence consumers and sync offsets to the destination

Step 2 moves consumer group coordination, including the committed offsets, to the destination cluster. The offsets must be on the destination cluster before that happens, and they must not change while you copy them. To stop them from changing, fence the entire consume plane first.

Note

To defer the offset sync, set coordination.group to the source cluster in step 2, and skip this step. Sync the offsets to the destination after you migrate all topics, as step 8 describes.

The spec.hotReload setting controls whether Confluent Gateway applies each change in this procedure without restarting its pods or CFK rolls the Confluent Gateway pods:

  • If spec.hotReload.enabled is true and you use a license other than the Trial license, Confluent Gateway applies changes to routes, including rules, without rolling its pods. By default, spec.hotReload.enableCanary is true, so Confluent Gateway validates each change in a canary pod before it applies the change to the other pods.

    kind: Gateway
spec:
  hotReload:
    enabled: true

  • If spec.hotReload.enabled is false, which is the default, CFK rolls the Confluent Gateway pods to apply each change.

  • In Trial mode, Confluent Gateway can’t apply changes without a restart. If hot reload is enabled, CFK doesn’t roll the pods, so your changes don’t take effect until you restart them. Keep hot reload disabled and let CFK roll the pods.

If you don’t use the Trial license, enable hot reload now, because every step in this procedure applies the CR. With hot reload enabled, fencing and rerouting don’t roll the pods. Applying a change to a route replaces the route and drops all of its client connections. The clients reconnect.

  1. Block all consume traffic on the route that your clients use. Add the following fencing rule, which has no topics, to the rules section of the route in your CR file. If your clients already connect through a static route, add the rule to that route.

    kind: Gateway
spec:
  routes:
    - name: migration-route
      rules:
        fencing:
          - trafficType: CONSUME
            blocked: true

    A rule without topics also blocks the JoinGroup, SyncGroup, and Heartbeat requests, which don’t name a topic. A rule that lists topics doesn’t block them. Consumers can’t change offsets while the fence is in place. Producers aren’t affected. If the route uses the deprecated fence setting, replace it with rules.fencing, because a route can’t use both.

  2. Apply the CR:

    kubectl apply -f <gateway-CR-file>

  3. Wait until every Confluent Gateway pod has applied the change. With hot reload, each pod picks up the change at its own time, so check the log of each pod for the Hot-reload successful message. Without hot reload, wait for the pods to roll, for example with kubectl rollout status. Until every pod has the fence, a pod without it still lets consumers commit offsets on the source cluster, and the copy can miss those commits.

  4. Sync the consumer group offsets from the source cluster to the destination cluster, for example with consumer offset sync on the cluster link. Confirm that the offsets on the destination match the source.

  5. Disable consumer offset sync on the cluster link, and on any other replicator, so that it can’t overwrite the offsets that clients commit through Confluent Gateway afterward.

Step 2: Configure a dynamic route and remove the consumer fence

Make the following changes in a single update: switch to the dynamic route with coordination.group set to the destination, and remove the consumer fence rule from step 1. Consumers then reconnect and find their coordinator on the destination cluster.

Bind the cp and cc streaming domains to the route.

  • Set cp, the source cluster, as the default owner of all topics. Clients continue to reach every topic through the same bootstrap address.

  • If your clients already connect through a static route, update that route instead of adding a new one.

  • Don’t set the fence setting on the route, because a route can’t use fence and rules.fencing together.

For more information, see Fencing filter.

kind: Gateway
spec:
  routes:
    - name: migration-route
      endpoint: migration-gw.example.com:9092
      mode: DYNAMIC
      brokerIdentificationStrategy:
        type: host
        pattern: broker-$(nodeId).migration-gw.example.com:9092
      streamingDomains:
        - name: cp
          bootstrapServerId: cp-bootstrap
        - name: cc
          bootstrapServerId: cc-bootstrap
      security:
        client:
          tls:
            secretRef: gateway-tls
      rules:
        routing:
          coordination:
            group: cc
          default: cp

This example shows only the settings that topic-based routing requires. You must also define the cp and cc streaming domains in spec.streamingDomains. For more information, see Configure streaming domains.

Apply the CR:

kubectl apply -f <gateway-CR-file>

Set coordination.group to the destination, cc, so that Confluent Gateway coordinates consumer groups and their committed offsets on that cluster. Every topic that the groups read must exist there.

For all routing settings, see Configure routing rules (Early Access).

Step 3: Check the mirror topic on the destination cluster

Confirm that the mirror topic for orders exists on the destination cluster and is replicating. The prerequisites require mirror topics for every topic that the consumer groups read. If the mirror topic doesn’t exist, use the cluster link to create it. For more information, see Mirror topics on Confluent Platform or Mirror topics on Confluent Cloud.

Confluent Gateway continues to send all orders traffic to the source cluster while the data replicates.

Step 4: Fence produce requests to the topic

Add a fencing rule that blocks produce requests to orders. Fencing produce requests stops new writes so that replication can catch up. Consumers can keep reading the data that already exists.

Producers receive an error that they can retry, and they keep retrying until you remove the fence or their delivery.timeout.ms expires. The default is 120,000 ms, which is two minutes. When the timeout expires, the producer stops retrying and returns an error to the application. To keep the fence short, remove it as soon as you update the routing in step 6. If you expect the fence to last longer than the timeout, increase delivery.timeout.ms for your producers before you fence the topic.

Add the fencing list to the rules section of the route in your CR file, as the following example shows. The other route settings don’t change. If you already moved other topics, keep their conditions in rules.routing.

kind: Gateway
spec:
  routes:
    - name: migration-route
      rules:
        routing:
          coordination:
            group: cc
          default: cp
        fencing:
          - trafficType: PRODUCE
            topics: ["orders"]
            blocked: true

Apply the CR:

kubectl apply -f <gateway-CR-file>

For more information about fencing rules and the errors that clients receive, see Configure fencing rules.

Step 5: Promote the mirror topic

Before you promote the mirror topic, confirm that every Confluent Gateway pod has applied the produce fence from step 4, as described in step 1. A pod that doesn’t have the fence yet keeps writing to the source cluster after the promotion, and those writes never reach the destination cluster.

After the mirror topic catches up with the source topic, promote the mirror topic on the destination cluster. For information about how to check the replication lag and promote a mirror topic, see Mirror topics on Confluent Platform or Mirror topics on Confluent Cloud.

Step 6: Update the routing and remove the fence

In the rules section of the route in your CR file, make the following changes:

  • Add a condition to rules.routing that routes orders to the destination cluster, cc.

  • Remove the fencing entry for orders. If rules.fencing has no other rules, remove the entire list.

Confluent Gateway evaluates conditions before it applies the default streaming domain, so orders goes to cc and all other topics stay on cp.

The following example shows only the rules section of the route:

kind: Gateway
spec:
  routes:
    - name: migration-route
      rules:
        routing:
          coordination:
            group: cc
          conditions:
            - topics: ["orders"]
              streamingDomain: cc
          default: cp

Apply the CR:

kubectl apply -f <gateway-CR-file>

Producers resume writing, and Confluent Gateway sends all orders traffic to the destination cluster.

Warning

Change the routing and remove the fence in a single CR change. If you remove the fence first, producers write to the source cluster after you promoted the mirror topic.

For a short time after you apply the change, clients that still have metadata for the source cluster might receive NOT_LEADER_OR_FOLLOWER errors. This is expected, because Confluent Gateway rejects requests to the topic on the source cluster. Clients refresh their metadata and retry the request.

Step 7: Verify the migration

  • On the destination cluster, the producer and consumer bandwidth for orders increases.

  • The rate of gateway_fencing_v2_requests_total for the orders topic is non-zero while the fence is active, and drops to zero after you remove the fence.

  • The rate of gateway_router_rejections_total with reason="cross-cluster-ownership" drops to zero after clients refresh their metadata.

  • gateway_router_metadata_topics_dropped_total with reason="owned-elsewhere" stops increasing.

  • gateway_router_fanout_leg_failures_total doesn’t increase.

Confluent Gateway serves the metrics on the admin port 9190 by default. To read them from a Confluent Gateway pod, forward the port and request the metrics endpoint:

kubectl port-forward pod/<gateway-pod-name> 9190:9190

curl -s http://localhost:9190/metrics | grep gateway_router

For more information about these metrics and other metrics to monitor, see Monitor topic-based routing and fencing in the Confluent Private Cloud Gateway metrics reference.

Step 8: Complete the migration

Repeat step 3 through step 7 for each remaining topic or group of topics. Keep the conditions for the topics that you already moved.

To migrate more than one topic at once, list the topics in topics, or match them with topicPatterns, in both the fencing rule in step 4 and the routing condition in step 6. For more information, see Configure fencing rules and Configure routing rules (Early Access).

After all topics have moved, the route no longer needs topic-based routing. Convert it back to a static route on the destination cluster, and remove rules.routing entirely. Complete the following steps:

  1. List the topics on the source cluster and confirm that each one has a routing condition that points to the destination streaming domain. Topics that clients created during the migration use the default streaming domain, which is the source cluster. Migrate any topics that are left first.

  2. If coordination.group points to the source cluster, move the consumer group coordinator first, in the same order as step 1: fence the consume plane, wait until every Confluent Gateway pod has applied the fence, and copy the committed offsets of all consumer groups to the destination cluster. Copy the offsets yourself, and include the offsets of every promoted topic, because Cluster Linking doesn’t sync offsets for promoted topics. If coordination.group already points to the destination, skip this step, because the static route keeps the same coordinator.

  3. In a single CR change, convert the route to a static route:

    • Replace streamingDomains with streamingDomain that references the destination streaming domain by name and bootstrapServerId.

    • Remove mode: DYNAMIC and the entire rules.routing section, including conditions, default, and coordination.

    • Remove any remaining rules.fencing entries, including the consume-plane fence from the previous step.

    • Keep the security.cluster settings only for the destination streaming domain, if you configured any.

  4. Apply the CR:

    kubectl apply -f <gateway-CR-file>

    Clients reconnect to the static route without changing their bootstrap address.

Warning

Don’t retire the source cluster until you apply the CR that unbinds it from the route. If you retire a cluster before you unbind it, requests that Confluent Gateway sends to every bound cluster fail.

Next steps