Migrate between Kafka Clusters using Confluent Gateway

Use the Client Switchover feature to migrate workloads from a source Apache Kafka® cluster to a destination cluster. Confluent Gateway acts as an intelligent proxy that handles all traffic routing and authentication transparently. This enables client applications to transition to new infrastructure automatically, without requiring any updates to existing client configurations.

Use cases for Client Switchover include:

Enable disaster recovery by switching from an unhealthy cluster to a healthy cluster.
Enable easier on-premises broker upgrades by enabling blue-green deployment strategies.

Tip

Client Switchover moves an entire route to another cluster at once. To move individual topics or groups of topics to another cluster, see Migrate Topics Incrementally with Topic-Based Routing and Fencing (Early Access).

Client Switchover architecture

Client Switchover routes traffic through a single Confluent Gateway route whose target Streaming Domain can be reconfigured to point to either the source or the destination Kafka cluster. Clients connect only to Confluent Gateway. The cluster swap happens server-side when the route configuration is updated.

Prerequisites

Before performing Client Switchover, set up data replication between the source and destination Kafka clusters outside of Confluent Gateway, using tools like Cluster Linking.

Considerations

Client Switchover has one hard limitation and a set of restart-related risks you can mitigate.

Hard limitation: Switching between clusters breaks message ordering and end-to-end consistency, because the source and destination are separate Kafka clusters. Don’t use Client Switchover for applications that require strict ordering, such as Kafka Streams applications.

Restart-related risks: When Confluent Gateway restarts to apply a Client Switchover, clients are affected differently depending on their role. Confluent Gateway restarts if you run it in Trial mode, or if the change touches a setting that Confluent Gateway can’t reload in place. With a Confluent Enterprise license for Confluent Platform that includes the Confluent Gateway add-on, Confluent Gateway watches its configuration file and applies most changes without a restart (hot reload). Hot reload replaces the changed route, so its client connections drop and the clients reconnect. The following table details the per-component impact of a restart and the mitigations:

Component	Impact during restart	Mitigation
Consumer	Consumer group rebalancing triggers if the restart exceeds `session.timeout.ms` (default `45` seconds). Consumers might reprocess messages if offsets weren’t committed before restart.	Set `isolation.level=read_committed` to avoid reading aborted transactions. Tune `group.protocol` (`classic` or, for static routes only, `consumer`), `partition.assignment.strategy`, and `session.timeout.ms` as needed. When `group.protocol` is `consumer`, Kafka ignores `session.timeout.ms`. Use the broker configuration `group.consumer.session.timeout.ms` instead.
Producer	Message duplication can occur when the broker acknowledges a message but Confluent Gateway fails before relaying the acknowledgment. This causes producer retries.	Set `enable.idempotence=true` to prevent duplicates, or ensure the application can handle them.
Transactions	Exactly Once Semantics (EOS) holds. In-flight transactions commit if the outage is shorter than `transaction.timeout.ms`, and otherwise abort. Any open transaction on the source aborts and cannot be committed on the target.	The producer application must reset the transaction state on failure and start a new transaction.

Perform Client Switchover

To switch clients from a source Kafka cluster to a destination Kafka cluster:

Configure Confluent Gateway with two Streaming Domains, one for each Kafka cluster. For more information, see Streaming domains configuration.

For example:

streamingDomains:
  - name: kafka1-domain
    type: kafka
    kafkaCluster:
      name: kafka-cluster-1
      bootstrapServers:
        - id: internal-plaintext-listener
          endpoint: "kafka-1:44444"
  - name: kafka2-domain
    type: kafka
    kafkaCluster:
      name: kafka-cluster-2
      bootstrapServers:
        - id: internal-plaintext-listener
          endpoint: "kafka-2:22222"

Define a switchover route that points to the source Kafka cluster. For example, the following route points to kafka1-domain, so clients send and receive messages from kafka-cluster-1:

routes:
  - name: switchover-route
    endpoint: "host.docker.internal:19092"
    streamingDomain:
      name: kafka1-domain
      bootstrapServerId: internal-plaintext-listener

To switch clients over to the destination Kafka cluster, update the streamingDomain.name in the route to kafka2-domain:

routes:
  - name: switchover-route
    endpoint: "host.docker.internal:19092"
    streamingDomain:
      name: kafka2-domain
      bootstrapServerId: internal-plaintext-listener

Apply the updated route.
- If you run Confluent Gateway with a Confluent Enterprise license for Confluent Platform that includes the Confluent Gateway add-on and a mounted configuration file, Confluent Gateway detects the change and reloads the configuration without a restart. To confirm, check the Confluent Gateway logs for Hot-reload successful. If the change touches a setting that can’t be reloaded in place, Confluent Gateway keeps the current configuration and logs that a restart is required.
- In Trial mode, or if Confluent Gateway keeps the current configuration, restart the container so that it loads the updated route:
```
docker compose -f docker-compose.yaml up -d --force-recreate gateway
```
After restart, clients continue sending and receiving new messages from the destination Kafka cluster. No changes are required on the producer or consumer side.