Migrate from Kafka MirrorMaker to Replicator in Confluent Platform

Kafka MirrorMaker is a standalone tool for copying data between two Apache Kafka® clusters. It is effectively a Kafka consumer and producer hooked together. MirrorMaker reads data from topics in the origin cluster and writes it to a topic with the same name in the destination cluster.

Important

Confluent Replicator is a more complete solution that copies topic configuration and data, and also integrates with Kafka Connect and Control Center to improve availability, scalability, and ease of use.

This topic provides examples of how to migrate from an existing datacenter that is using Kafka MirrorMaker to Replicator. In these examples, Replicator replicates messages from a specific timestamp, not from the beginning. This approach is helpful if you have a large number of legacy messages that you don’t want to migrate.

Assume there are two datacenters, DC1 (Active) and DC2 (Passive), and each is running a Kafka cluster. DC1 has a single topic, which is replicated to DC2 with the same topic name. The topic name is inventory.

Example 1: Same number of partitions in DC1 and DC2

In this example, you migrate from MirrorMaker to Replicator and keep the same number of partitions for inventory in DC1 and DC2.

Prerequisites:

• Confluent Platform 5.0.0 or later is installed.

• You must have the same number of partitions for inventory in DC1 and DC2 to use this method.

• The src.consumer.group.id in Replicator must match group.id in MirrorMaker.

1. Stop the running MirrorMaker instance in DC1, where <mm_pid> is the MirrorMaker process ID:

   kill <mm_pid>
   

2. Configure and start Replicator. In this example, Replicator runs as an executable from the command line or from a Docker image.

   1. Add these values to $CONFLUENT_HOME/etc/kafka-connect-replicator/replicator_consumer.properties. Replace localhost:9082 with the bootstrap.servers of DC1, the source cluster:

      bootstrap.servers=localhost:9082
      topic.preserve.partitions=true
      

   2. Add this value to CONFLUENT_HOME/etc/kafka-connect-replicator/replicator_producer.properties. Replace localhost:9092 with the bootstrap.servers of DC2, the destination cluster:

      bootstrap.servers=localhost:9092
      

   3. Ensure the replication factors are set to 2 or 3 for production, if they are not already:

      echo "confluent.topic.replication.factor=3" >> $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      echo "offset.storage.replication.factor=3" >> $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      echo "config.storage.replication.factor=3" >> $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      echo "status.storage.replication.factor=3" >> $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      

   4. Start Replicator:

      replicator --cluster.id <new_cluster_id> \
      --producer.config replicator_producer.properties \
      --consumer.config replicator_consumer.properties \
      --replication.config $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      

Replicator uses the offsets committed by MirrorMaker from DC1 and starts replicating messages from DC1 to DC2 based on these offsets.

Example 2: Different number of partitions in DC1 and DC2

In this example, you migrate from MirrorMaker to Replicator and have a different number of partitions for inventory in DC1 and DC2.

Prerequisites:

• Confluent Platform 5.0.0 or later is installed.

• The src.consumer.group.id in Replicator must match group.id in MirrorMaker.

1. Stop the running MirrorMaker instance from DC1.

2. Configure and start Replicator. In this example, Replicator runs as an executable from the command line or from a Docker image.

   1. Add this value to $CONFLUENT_HOME/etc/kafka-connect-replicator/replicator_consumer.properties. Replace localhost:9082 with the bootstrap.servers of DC1, the source cluster:

      bootstrap.servers=localhost:9082
      topic.preserve.partitions=false
      

   2. Add this value to CONFLUENT_HOME/etc/kafka-connect-replicator/replicator_producer.properties. Replace localhost:9092 with the bootstrap.servers of DC2, the destination cluster:

      bootstrap.servers=localhost:9092
      

   3. Ensure the replication factors are set to 2 or 3 for production:

      echo "confluent.topic.replication.factor=3" >> $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      echo "offset.storage.replication.factor=3" >> $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      echo "config.storage.replication.factor=3" >> $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      echo "status.storage.replication.factor=3" >> $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      

   4. Start Replicator:

      replicator --cluster.id <new_cluster_id> \
      --producer.config replicator_producer.properties \
      --consumer.config replicator_consumer.properties \
      --replication.config $CONFLUENT_HOME/etc/kafka-connect-replicator/quickstart-replicator.properties
      

Replicator uses the offsets committed by MirrorMaker from DC1 and starts replicating messages from DC1 to DC2 based on these offsets.

Next steps

• Sign up for Confluent Cloud and use the Cloud quick start to get started.

• Download Confluent Platform and use the Confluent Platform quick start to get started.