Migrate Topics Incrementally with Topic-Based Routing and Fencing

Using dynamic routing capabilities, a single Confluent Gateway route can route topics to different Apache Kafka® clusters at the same time, so you can migrate topics without updating client bootstrap addresses or restarting applications.

These clusters can be any Kafka cluster, including Confluent Platform, Confluent Cloud, and open-source Kafka. You can migrate topics to a new cluster individually or in batches.

This guide explains how to use topic-based fencing and routing to migrate a batch of topics without disrupting clients. Before you begin, review the limitations.

Important

Dynamic routing, which topic-based routing uses, is an Early Access Program feature in Confluent Gateway 1.4. Fencing is generally available.

An Early Access feature is a component of Confluent Cloud introduced to gain feedback. This feature should be used only for evaluation and non-production testing purposes or to provide feedback to Confluent, particularly as it becomes more widely available in follow-on preview editions.

Early Access Program features are intended for evaluation use in development and testing environments only, and not for production use. Early Access Program features are provided: (a) without support; (b) “AS IS”; and (c) without indemnification, warranty, or condition of any kind. No service level commitment will apply to Early Access Program features. Early Access Program features are considered to be a Proof of Concept as defined in the Confluent Cloud Terms of Service. Confluent may discontinue providing preview releases of the Early Access Program features at any time in Confluent’s sole discretion.

Configuration schemas for dynamic routing, including property names and YAML structure, are subject to change before General Availability (GA) and might not be backward compatible.

What topic-based routing and fencing do

A static Confluent Gateway route sends all traffic to a single Kafka cluster. A dynamic route that defines topic-based rules can serve topics from more than one cluster. Fencing pauses traffic for selected topics at the topic level, in a way that clients retry, while you use Cluster Linking or Replicator to move the topics.

Use topic-based routing to:

Migrate gradually. You move topics to the new cluster in batches, instead of cutting over all the topics at once.
Keep clients unchanged. Clients connect to one bootstrap address, and Confluent Gateway sends each request to the cluster that owns the topic.
Avoid writes to both clusters. Fencing pauses writes to a topic while its data finishes replicating, and producers retry the fenced requests. Fencing can also pause reads if you need it to.

Topic-based routing differs from Client Switchover, which moves an entire route to another cluster at once.

How topic-based routing and fencing work

Confluent Gateway models each Kafka cluster as a streaming domain (Streaming domains configuration). Topic-based routing and fencing build on this model and use three main components:

Component

Description

Configuration reference

Dynamic route

A route that uses topic-based routing. It binds more than one streaming domain using streamingDomains to serve topics from more than one cluster through one route.

Routes configuration

Routing rules (rules.routing)

Maps topics to a streaming domain by exact name or pattern. Confluent Gateway checks the rules in order, and the first match wins. Unmatched topics fall back to the default streaming domain.

Routing rules configuration (Early Access)

Fencing rules (rules.fencing)

Pauses or allows traffic for selected topics. Confluent Gateway checks the rules in order, and the first match wins. Blocked requests return errors that clients can retry, or empty responses, so most clients retry instead of failing. Fencing rules also work on static routes.

Fencing rules configuration

Responses to blocked requests

Migration sequence

Because only one cluster owns a topic at a time, migrating a topic requires the replicate-fence-promote-reroute workflow:

Replicate: Set up replication to the destination cluster.
Fence: Pause writes to the topics that you are about to migrate.
Promote: Promote the mirror topic on the destination cluster. Make sure that replication has caught up first.
Reroute: Update the routing rules to point to the new cluster, and remove the fence.

For the step-by-step procedure, see Migrate topics between clusters.

Limitations

Before you plan a migration, review the following limitations of topic-based routing and fencing.

A dynamic route has the following limitations:

A dynamic route supports only the host broker identification strategy. The host strategy requires tls configuration.
A dynamic route doesn’t support transactions or queues for Kafka (share groups). Confluent Gateway advertises these APIs as unsupported. It rejects requests from transactional producers with an INVALID_REQUEST error, which is fatal to the client, so transactional producers fail instead of retrying.
Before you convert a static route to a dynamic route, make sure that no transactional clients use the route, including Kafka Streams applications and Kafka Connect workers in exactly-once mode. Transactional producers that are running on the route when you convert it fail in the middle of a transaction. The broker aborts the open transaction only after transaction.timeout.ms expires, and read_committed consumers of the affected partitions can’t read past it until then.
Confluent Gateway doesn’t support the following APIs on a dynamic route. Unsupported requests return an UNSUPPORTED_VERSION error.
- Consumer group management APIs, such as DescribeGroups, ListGroups, DeleteGroups, and OffsetDelete. As a result, the kafka-consumer-groups tool doesn’t work on a dynamic route.
- Admin management APIs, such as the access control list (ACL) APIs (CreateAcls and DeleteAcls), DescribeConfigs, AlterConfigs, CreatePartitions, and DeleteRecords.
A dynamic route doesn’t support the consumer group protocol. A consumer with group.protocol=consumer fails at startup with a fatal exception, because Confluent Gateway doesn’t advertise the ConsumerGroupHeartbeat API. Set group.protocol=classic explicitly on your consumers, because Apache Kafka® plans to change the default in 5.0. A group that already uses the consumer protocol converts back to the classic protocol when it’s empty, so switch all of its members at once.
A dynamic route doesn’t support topic auto-creation. Clients must create topics with the CreateTopics API.
The clusters that a dynamic route binds can run different Kafka versions. For example, one cluster can run 3.9 and another can run 4.0.
When all authentication legs of a route use passthrough, Confluent Gateway sends the client’s Simple Authentication and Security Layer (SASL) credential to every bound cluster, so every cluster must accept the same credential.
Salted Challenge Response Authentication Mechanism (SCRAM) client authentication requires that you configure authentication swap for every cluster. Confluent Gateway doesn’t support SCRAM with passthrough, or for authenticating to a cluster. Use PLAIN or OAUTHBEARER instead. To configure authentication on a route, see Route authentication configuration.

Fencing has the following limitations:

Fencing lets you pause only PRODUCE or CONSUME traffic for selected topics. For details, see fencing rules. Confluent Gateway doesn’t fence other operations, such as fetching metadata or creating and deleting topics.
Confluent Gateway doesn’t fence offset fetches.
A producer that uses acks=0 can’t receive an error response, so Confluent Gateway silently drops the records for fenced topics. Use acks=1 or acks=all for the topics that you migrate.

Migrate topics between clusters

Migrate topics between Kafka clusters in a Docker-based Confluent Gateway deployment without client downtime or application restarts.

To migrate topics in a Confluent for Kubernetes deployment, see Migrate Topics with Topic-Based Routing and Fencing using Confluent for Kubernetes.

Example setup

The migration sequence uses the following example environment:

Topic: orders
Source cluster: cp streaming domain (Confluent Platform)
Destination cluster: cc streaming domain (Confluent Cloud)

Note

This sequence applies to any set of topics and to any pair of clusters that Cluster Linking supports. The destination cluster must be able to host mirror topics.

Prerequisites

Before you begin, make sure that you have:

Confluent Gateway 1.4, deployed with Docker. For more information, see Configure and Deploy Confluent Cloud Gateway.
A streaming domain defined for each cluster. For more information, see Streaming domains configuration.
A cluster link and mirror topics on the destination cluster, created outside Confluent Gateway. The destination cluster needs a mirror topic for every topic that the consumer groups of the route read. Create them before step 1, or enable auto-create mirror topics on the cluster link. Step 2 moves consumer group coordination to the destination cluster. The destination cluster rejects offset commits for topics that it doesn’t have. For more information, see Manage Mirror Topics for Cluster Linking on Confluent Cloud.
Client applications that connect to Confluent Gateway through a route. If your clients already connect through a static route, you convert that route in step 2. A dynamic route supports only the host broker identification strategy, which requires an SSL keystore in the route’s security settings. If your existing route uses the port strategy, clients must change to the new broker addresses first.
All members of each consumer group connect to Confluent Gateway through the route at the same time. If some members still connect to the source cluster directly while coordination.group is the destination, the group is split across two coordinators and the members process partitions twice.
A fence can’t pause clients that connect to the source cluster directly. For those clients, stop all members of each consumer group, sync the offsets, and then start the members on the route.
Producers whose delivery.timeout.ms is longer than the time that you expect a topic to stay fenced. For more information, see step 4.

Prepare authentication

With authentication swap, clients authenticate to Confluent Gateway with one set of credentials, and Confluent Gateway replaces them with its own credentials when it connects to each cluster. Configure clientAuth once for the clients, and clusterAuth for each streaming domain. A route that uses SCRAM client authentication requires authentication swap for every cluster.

Choose and configure a secret store that holds the credentials that Confluent Gateway uses to connect to each cluster. Use the secret store provider that fits your environment, such as AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, or CyberArk Conjur. Add the store to gateway.secretStores and note its name, because the secretStore setting in the security section that you add later in this procedure must use it.

Store one secret for each client user. Confluent Gateway looks up the secret by the client’s username, and uses the username and password in it to connect to the cluster. Providers differ in how they locate and format a secret, for example in prefixPath, separator, and useJson. For the settings and the secret format of your provider, see Secret store configuration.
Create the Java Authentication and Authorization Service (JAAS) file for client authentication, for example /etc/gateway/client-plain-jaas.conf. It lists the users that connect to Confluent Gateway, with one user_<username> property for each user. For example:
```
org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-secret"
    user_admin="admin-secret"
    user_additional-user="additionaluser-secret";
```
For more information, see Client authentication for authentication swapping.
Create a JAAS template file for each cluster, for example /etc/gateway/cp-swap-jaas.conf and /etc/gateway/cc-swap-jaas.conf. Keep the %s placeholders as they are. Confluent Gateway fills them with the username and password that it reads from the secret store.
```
org.apache.kafka.common.security.plain.PlainLoginModule required username="%s" password="%s";
```
For more information, see Cluster authentication for authentication swapping.
Mount the three JAAS files, and any other file that your secret store configuration references, into the Confluent Gateway container at the paths that you set in the configuration, for example with a volumes entry in docker-compose.yaml. For the settings of your secret store, see Secret store configuration.

Add the security section to the route. The following example shows only the security section of the route:

gateway:
  routes:
    - name: migration-route
      security:
        clientAuth:
          sasl:
            mechanism: PLAIN
            callbackHandlerClass: org.apache.kafka.common.security.plain.internals.PlainServerCallbackHandler
            jaasConfig:
              file: /etc/gateway/client-plain-jaas.conf
        clusterAuth:
          cp:
            auth: swap
            secretStore: kms-store
            sasl:
              mechanism: PLAIN
              callbackHandlerClass: org.apache.kafka.common.security.authenticator.SaslClientCallbackHandler
              jaasConfig:
                file: /etc/gateway/cp-swap-jaas.conf
          cc:
            auth: swap
            secretStore: kms-store
            sasl:
              mechanism: PLAIN
              callbackHandlerClass: org.apache.kafka.common.security.authenticator.SaslClientCallbackHandler
              jaasConfig:
                file: /etc/gateway/cc-swap-jaas.conf

The example uses kms-store as the secret store name. Replace it with the name of the secret store that you configured earlier in this section. Each key under clusterAuth must match the name of a streaming domain that the route binds, which in this example is cp or cc.

Keep the security section ready. You add it to the route in step 2, so that the authentication settings take effect in the same configuration change as the dynamic route. For all authentication settings, see Route authentication configuration.

Step 1: Fence consumers and sync offsets to the destination

Step 2 moves consumer group coordination, including the committed offsets, to the destination cluster. The offsets must be on the destination cluster before that happens, and they must not change while you copy them. To stop them from changing, fence the entire consume plane first.

Note

To defer the offset sync, set coordination.group to the source cluster in step 2, and skip this step. Sync the offsets to the destination after you migrate all topics, as step 8 describes.

You apply every configuration change in this guide the same way. How you apply a change depends on how you provide the configuration:

If the configuration is inline in docker-compose.yaml, as in the GATEWAY_CONFIG environment variable, Confluent Gateway reads it only at startup. Recreate the container to apply a change:
```
docker compose -f docker-compose.yaml up -d
```
If you mount the configuration file and set GATEWAY_CONFIG_FILE, Confluent Gateway applies route and rules changes without a restart. This doesn’t apply in Trial mode. In Trial mode, restart Confluent Gateway to apply the changes.

Applying a change to a route replaces the route and drops all of its client connections. The clients reconnect. When Confluent Gateway applies a change without a restart, it rejects an invalid change and keeps the previous configuration, so check the Confluent Gateway log after you apply a change. If you restart Confluent Gateway, an invalid configuration stops it from starting.

Block all consume traffic on the route that your clients use. Add the following fencing rule, which has no topics, and apply the configuration. If your clients already connect through a static route, add the rule to that route.
```
gateway:
  routes:
    - name: migration-route
      rules:
        fencing:
          - trafficType: CONSUME
            blocked: true
```
A rule without topics also blocks the JoinGroup, SyncGroup, and Heartbeat requests, which don’t name a topic. A rule that lists topics doesn’t block them. Consumers can’t change offsets while the fence is in place. Producers aren’t affected. If the route uses the deprecated fence setting, replace it with rules.fencing, because a route can’t use both.
Sync the consumer group offsets from the source cluster to the destination cluster, for example with consumer offset sync on the cluster link. Confirm that the offsets on the destination match the source.
Disable consumer offset sync on the cluster link, and on any other replicator, so that it can’t overwrite the offsets that clients commit through Confluent Gateway afterward.

Step 2: Configure a dynamic route and remove the consumer fence

Make the following changes in a single configuration change: switch to the dynamic route with coordination.group set to the destination, add the security section that you prepared, and remove the consumer fence rule from step 1. Consumers then reconnect and find their coordinator on the destination cluster.

Bind both streaming domains to the route. Set the source cluster as the default owner of all topics. Clients continue to reach every topic through the same bootstrap address. If your clients already connect through a static route, update that route instead of adding a new one. Use rules.fencing instead of the deprecated fence setting, because a route can’t use both. For more information, see Fencing filter.

gateway:
  routes:
    - name: migration-route
      endpoint: migration-gw.example.com:9092
      mode: DYNAMIC
      brokerIdentificationStrategy:
        type: host
        pattern: broker-$(nodeId).migration-gw.example.com:9092
      streamingDomains:
        - name: cp
          bootstrapServerId: cp-bootstrap
        - name: cc
          bootstrapServerId: cc-bootstrap
      rules:
        routing:
          coordination:
            group: cc
          default: cp

This example shows only the settings that topic-based routing requires. A route that uses type: host also requires an SSL keystore in its security settings. For more information, see Configure Security for Confluent Cloud Gateway.

Add the security section that you prepared to the same route, and then apply the configuration the same way that you did in step 1.

The coordination.group setting chooses the streaming domain that coordinates all consumer groups on the route, including their committed offsets. This example uses the destination, cc. Every topic that the groups read must exist on that cluster, because the cluster rejects offset commits for topics that it doesn’t have.

For all routing settings, see Routing rules configuration (Early Access).

Step 3: Check the mirror topic on the destination cluster

Confirm that the mirror topic for orders exists on the destination cluster and is replicating. If it doesn’t exist, use the cluster link to create it.

Confluent Gateway continues to send all orders traffic to the source cluster while the data replicates.

Step 4: Fence produce requests to the topic

Add a fencing rule that blocks produce requests to orders. Fencing produce requests stops new writes so that replication can catch up. Consumers can keep reading the data that already exists.

Producers receive an error that they can retry, and they keep retrying until you remove the fence or their delivery.timeout.ms expires. The default is 120,000 ms, which is two minutes. When the timeout expires, the producer stops retrying and returns an error to the application. To keep the fence short, remove it as soon as you update the routing in step 6. If you expect the fence to last longer than the timeout, increase delivery.timeout.ms for your producers before you fence the topic.

Add the fencing list to the rules section of the route, as the following example shows. The other route settings don’t change. If you already moved other topics, keep their conditions in rules.routing.

gateway:
  routes:
    - name: migration-route
      rules:
        routing:
          coordination:
            group: cc
          default: cp
        fencing:
          - trafficType: PRODUCE
            topics: ["orders"]
            blocked: true

Apply the configuration to Confluent Gateway the same way that you did in step 1. For more information about fencing rules and the errors that clients receive, see Fencing rules configuration.

Step 5: Promote the mirror topic

After the mirror topic catches up with the source topic, promote the mirror topic on the destination cluster. For information about how to check the replication lag and promote a mirror topic, see Manage Mirror Topics for Cluster Linking on Confluent Cloud.

Step 6: Update the routing and remove the fence

Add a condition that routes orders to the destination cluster. Remove the fencing entry for orders, or the entire rules.fencing list if it has no other rules. Confluent Gateway evaluates conditions before it applies the default streaming domain. As a result, orders goes to cc, and all other topics stay on cp.

The following example shows only the rules section of the route:

gateway:
  routes:
    - name: migration-route
      rules:
        routing:
          coordination:
            group: cc
          conditions:
            - topics: ["orders"]
              streamingDomain: cc
          default: cp

Apply the updated configuration to Confluent Gateway, as in step 1. Producers resume writing, and Confluent Gateway sends all orders traffic to the destination cluster.

Warning

Change the routing and remove the fence in a single configuration change. If you remove the fence first, producers write to the source cluster after you promoted the mirror topic.

Confluent Gateway returns a NOT_LEADER_OR_FOLLOWER error, which clients can retry, to clients that still hold metadata that points to the source cluster. Instead of reading from or writing to the source topic, the clients refresh their metadata.

Step 7: Verify the migration

Confirm that producers and consumers of orders work without any changes to their configuration. Then check the producer and consumer bandwidth of the Kafka clusters for the orders topic. After the migration, the bandwidth decreases on the source cluster and increases on the destination cluster.

For other metrics that you can monitor on a route, see Monitor topic-based routing and fencing and Monitor Confluent Cloud Gateway Metrics.

Step 8: Complete the migration

Repeat step 3 through step 7 for each remaining topic or group of topics. Keep the conditions for the topics that you already moved.

To migrate more than one topic at once, list the topics in topics, or match them with topicPatterns, in both the fencing rule in step 4 and the routing condition in step 6. For more information, see Fencing rules configuration and Routing rules configuration (Early Access).

After all topics have moved, the route no longer needs topic-based routing. Convert it back to a static route on the destination cluster, and remove rules.routing entirely. Complete the following steps:

List the topics on the source cluster and confirm that each one has a routing condition that points to the destination streaming domain. Topics that clients created during the migration go to the default streaming domain, which is the source cluster. Migrate any topics that are left first.
If coordination.group points to the source cluster, move the consumer group coordinator first, in the same order as step 1: fence the consume plane, and copy the committed offsets of all consumer groups to the destination cluster. Copy the offsets yourself, and include the offsets of every promoted topic, because Cluster Linking doesn’t sync offsets for promoted topics. If coordination.group already points to the destination, skip this step, because the static route keeps the same coordinator.
In a single configuration change, convert the route to a static route:
- Replace streamingDomains with streamingDomain set to the destination streaming domain.
- Remove mode: DYNAMIC and the entire rules.routing section, including conditions, default, and coordination.
- Remove any remaining rules.fencing entries, including the consume-plane fence from the previous step.
- Optionally, keep clusterAuth only for the destination streaming domain.
Apply the configuration. Clients reconnect to the static route without changing their bootstrap address.

Warning

Don’t retire the source cluster until you apply the configuration that unbinds it from the route. If a retired cluster is still bound, requests that Confluent Gateway sends to every bound cluster fail.

Next steps

To monitor routes during a migration, see Monitor Confluent Cloud Gateway Metrics.
To move an entire route at once, see Migrate between Kafka Clusters using Confluent Cloud Gateway.