Migrate to Confluent Cloud

You can migrate your existing Apache Kafka® workloads to Confluent Cloud from sources such as a self-managed Kafka deployment or Amazon Web Services (AWS) Managed Streaming for Kafka (MSK). A typical migration involves these steps:

Replicate your data to Confluent Cloud.
Migrate your schemas so that clients can serialize and deserialize data on the new cluster.
Migrate your connectors to Confluent Cloud managed connectors.
Switch over your clients to the new cluster.
Validate your migration and decommission the source cluster.

The following sections describe each step.

Tip

Before you migrate, plan your migration in the Confluent Cloud Console Migration hub. The Migration hub provides a step-by-step migration guide with a checklist to help you plan, run, and validate your migration. You must sign in to access this resource.

Replicate your data

Choose a data replication tool to copy your topic data to Confluent Cloud.

Cluster Linking (recommended for most migrations): Makes a byte-for-byte copy of your topics on Confluent Cloud and synchronizes consumer offsets between the clusters. It is fully managed and requires no extra infrastructure. For more information, see Migrate data with Cluster Linking on Confluent Cloud. To link a self-managed or on-premises cluster to Confluent Cloud, see Link Self-Managed and Confluent Cloud Clusters for Hybrid Cloud and Bridge-to-Cloud Deployments.
Confluent Replicator: Replicates topics and can repartition or reconfigure them during the migration. Use Replicator when Cluster Linking doesn’t support your source and destination cluster pairing. For more information, see Use Replicator to Migrate Topic Data on Confluent Cloud and Example: Use Confluent Replicator to Copy Kafka Data to Confluent Cloud.
MirrorMaker 2: Replicates data, but has no managed option on Confluent Cloud, so use it only if you cannot use Cluster Linking or Replicator. For more information, see Kafka MirrorMaker.

Migrate your schemas

If your clients use a Schema Registry, migrate your schemas to Confluent Cloud so that producers and consumers can serialize and deserialize data on the new cluster. Use Schema Linking, a fully managed service that copies schemas and keeps them in sync between two registries. For more information, see Schema Linking on Confluent Cloud.

Migrate your connectors

Move your self-managed connectors to Confluent Cloud managed connectors so that your data pipelines continue to run after the migration. For more information, see Migrate Self-Managed Connectors to Fully-Managed Connectors.

Switch over your clients

Choose a method to switch over your client applications to Confluent Cloud.

Zero-cut migration (recommended for minimal downtime): Confluent Gateway routes traffic and maps authentication so clients keep their original endpoints and credentials and reconnect automatically at cutover, with no configuration changes. For MSK migrations, the Kafka Copy Paste (kcp) tool automates this process. For more information, see Migrate between Kafka Clusters using Confluent Cloud Gateway and Migrate Clusters with the Kafka Copy Paste (kcp) Tool.
Client restart: Update each client’s bootstrap server and security configuration and restart it against Confluent Cloud. Several migration patterns are available, such as restart all at once, stop-wait-restart, and stop-restart-repeat, depending on how much downtime and how many duplicate or missed messages your clients can tolerate. For the Cluster Linking cutover steps, see Migrate data with Cluster Linking on Confluent Cloud.

Important

Before you onboard clients to Confluent Gateway, make sure they use a supported authentication method. Clients that authenticate to MSK with AWS IAM must move to SASL/SCRAM or mTLS first.

Validate your migration

After you switch over your clients, confirm that your workloads run as expected on Confluent Cloud and meet your performance requirements. Then decommission the source cluster and any temporary infrastructure you created for the migration.