Replicator Quick Start to Migrate Topic Data on Confluent Cloud

You can use Replicator to copy or move topic data across Confluent Cloud clusters. Replicator runs in one of three modes: as a connector, as an executable on a virtual machine (VM), or as an executable on Kubernetes. This quick start uses Replicator as an executable on an Ubuntu VM to migrate topics between two Confluent Cloud clusters on Amazon Web Services (AWS).

If you migrate clusters from another cloud platform, such as Google Cloud or Microsoft Azure, the same general procedure applies.

Prerequisites

Two Confluent Cloud clusters on AWS: A source cluster and a destination cluster, including the bootstrap server and SASL credentials for each.
An Ubuntu VM in Amazon EC2.
The Confluent CLI installed.

Install Java and Confluent Platform

Install Java.
```
sudo apt-get install default-jre
```
Use APT to install the full Confluent Platform as described in Manual Install using Systemd on Ubuntu and Debian.

Configure Replicator properties files

Replicator uses three configuration files: one for the source cluster, one for the destination cluster, and one for the replication rules.

Source cluster

consumer.properties configures the connection to the source cluster. Replace bootstrap.servers and sasl.jaas.config with the values for your source cluster.

bootstrap.servers=<source bootstrap server>
ssl.endpoint.identification.algorithm=https
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
security.protocol=SASL_SSL

Destination cluster

producer.properties configures the connection to the destination cluster. Replace bootstrap.servers and sasl.jaas.config with the values for your destination cluster.

bootstrap.servers=<destination bootstrap server>
ssl.endpoint.identification.algorithm=https
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
security.protocol=SASL_SSL

Replication rules

replication.properties defines which topics to replicate and the rename rules. Replace “Movies” in topic.whitelist with the topics you want to replicate from the source cluster.

topic.whitelist=Movies

topic.rename.format=${topic}-replica
topic.auto.create=true
topic.timestamp.type=CreateTime

dest.topic.replication.factor=3

In the preceding example, topic.rename.format, topic.auto.create, and topic.timestamp.type are set to their default values, so you can omit them. Include a property only to override its default.

Run Replicator

Run the Replicator executable to migrate topics.

confluent-5.3.0/bin/replicator \
  --consumer.config ./diyrepl/consumer.properties \
  --producer.config ./diyrepl/producer.properties \
  --cluster.id replicator  \
  --replication.config ./diyrepl/replication.properties

Verify topic migration

To verify that Replicator migrated your topic data, use the Confluent CLI to read records from the source and destination topics, then confirm that new records produced on the source appear on the destination. Before you begin, log in to the Confluent CLI.

Check current destination topic contents

Read the destination topic to see the data that Replicator replicated from the source.

List the clusters and note the destination cluster ID.
```
confluent kafka cluster list
```

Select the destination cluster.

confluent kafka cluster use <ID-for-destination-cluster>

Run the consumer to read from your topic on the destination (for example, Movies-replica).
```
confluent kafka topic consume --from-beginning 'Movies-replica'
```

Check current contents of source topic

Read the source topic to see the original data.

Select the source cluster.

confluent kafka cluster use <ID-for-source-cluster>

Run the consumer to read from your topic on the source (for example, Movies).
```
confluent kafka topic consume --from-beginning 'Movies'
```

Produce a record to the source topic

To verify ongoing replication, produce a new record on the source topic, then re-check the destination topic.

confluent kafka topic produce Movies

Check logs and destination topic

Confirm the new record replicated by checking the Replicator logs and reading the destination topic again.

Check the output of the running Replicator task for logs indicating processing.