Replicator Quick Start to Migrate Topic Data on Confluent Cloud¶

You can to use Replicator copy or move topic data across Confluent Cloud clusters by running Replicator in one of three modes (connector, executable on a VM, or executable on Kubernetes).

This quick start was created with the following specifics and assumes:

You are migrating topics between two Confluent Cloud clusters on Amazon Web Services (AWS).
You are running Replicator on an Ubuntu VM in Amazon EC2.
You are running Replicator as an executable.

In real-world scenarios, you might migrate clusters from any other cloud platform nodes (for example, GCP and Google Cloud Console Google Cloud Console or Microsoft Azure). The same general procedure applies.

Set up a Cloud instance¶

Install Java.
```
sudo apt-get install default-jre
```
Use APT to install the full Confluent Platform as described in Manual Install using Systemd on Ubuntu and Debian.

Configure properties¶

There are three config files for consumer, producer, and replication. The minimal configuration changes for these are shown below.

Tip

Replace bootstrap.servers and sasl.jaas.config here with the corresponding values for the Confluent Cloud clusters.

consumer.properties

bootstrap.servers=<source bootstrap server>
ssl.endpoint.identification.algorithm=https
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
security.protocol=SASL_SSL

producer.properties

bootstrap.servers=<destination bootstrap server>
ssl.endpoint.identification.algorithm=https
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
security.protocol=SASL_SSL

replication.properties

Replace “Movies” for the topic.whitelist with the topics you want to replicate from the source cluster.

topic.whitelist=Movies

topic.rename.format=${topic}-replica
topic.auto.create=true
topic.timestamp.type=CreateTime

dest.topic.replication.factor=3

Tip

In the example above, topic.rename.format, topic.auto.create, and topic.timestamp.type are set to the defaults, and therefore not really necessary to include. If you want to change the values for these, include them in replication.properties with custom values.

Run Replicator¶

Run the Replicator executable to migrate topics.

confluent-5.3.0/bin/replicator \
  --consumer.config ./diyrepl/consumer.properties \
  --producer.config ./diyrepl/producer.properties \
  --cluster.id replicator  \
  --replication.config ./diyrepl/replication.properties

Verify topic migration¶

To verify topic migration is successful, log in to the Confluent CLI and run commands to: verify that your topic data has been replicated onto the target cluster.

After checking the current contents of source and destination clusters, perform the cutover of clients to the new cluster, and run consumers to read from the destination topic on the new cluster.

Check current contents of destination topic¶

List the clusters and note the destination cluster ID.
```
confluent kafka cluster list
```

Select the destination cluster.

confluent kafka cluster use <ID-for-destination-cluster>

Run the consumer to read from your topic on the destination (for example, Movies-replica).
```
confluent kafka topic consume --from-beginning 'Movies-replica'
```

Check current contents of source topic¶

Select the destination cluster.

confluent kafka cluster use <ID-for-source-cluster>

Run the consumer to read from your topic on the source (for example, Movies).
```
confluent kafka topic consume --from-beginning 'Movies'
```

Add something new to the source topic¶

Produce to the source topic.

confluent kafka topic produce Movies

Cutover to new cluster¶

Switch producers and consumers to the new cluster, as described in Cutover clients to new cluster.

Check logs and destination topic¶

Check output of running replicator task for logs indicating processing.

Run the consumer to read from your destination topic again.

confluent kafka topic consume --from-beginning 'Movies'

This concludes the Quick Start.

The following sections describe more general setup and tasks for using Replicator in any of its available modes to migrate topic data.