Confluent Replicator allows you to easily and reliably replicate topics from one Apache Kafka® cluster to another. In addition to copying the messages, this connector will create topics as needed preserving the topic configuration in the source cluster. This includes preserving the number of partitions, the replication factor, and any configuration overrides specified for individual topics.
The Replicator Connector supports replication in the context of several use cases, including:
- Multi-DC Deployment Architectures
- Multi-Datacenter Setup
- Migrate Schemas
- Networking in Confluent Cloud and other Cloud and hybrid scenarios
Replicator supports the following features:
- Topic selection using whitelists, blacklists, and regular expressions.
- Dynamic topic creation in the destination cluster with matching partition counts, replication factors, and topic configuration overrides.
- Automatic resizing of topics when new partitions are added in the source cluster.
- Automatic reconfiguration of topics when topic configuration changes in the source cluster.
- Starting with Confluent Platform 5.0.1, Replicator fully supports Timestamp Preservation, Provenance Headers to Prevent Cyclic Message Repetition, and Consumer Offset Translation.
- Starting with Confluent Platform 5.0.0, you can migrate from MirrorMaker to Replicator on existing datacenters. Migration from MirrorMaker to Replicator is not supported in earlier versions of Confluent Platform.
Install Replicator Connector¶
This connector is bundled natively with Confluent Platform. If you have Confluent Platform installed and running, there are no additional steps required to install.
If you are using Confluent Platform using only Confluent Community components, you can install the connector using the Confluent Hub client (recommended) or you can manually download the ZIP file.
Install the connector using Confluent Hub¶
- Confluent Hub Client must be installed. This is installed by default with Confluent Enterprise.
Navigate to your Confluent Platform installation directory and run the following command to install the latest (
latest) connector version. The connector must be installed on every machine where Connect will run.
confluent-hub install confluentinc/kafka-connect-|crep|:latest
You can install a specific version by replacing
latest with a version number. For example:
confluent-hub install confluentinc/kafka-connect-replicator:5.5.2
Install the connector manually¶
Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.
You can use this connector for a 30-day trial period without a license key.
After 30 days, this connector is available under a Confluent enterprise license. Confluent issues enterprise license keys to subscribers, along with providing enterprise-level support for Confluent Platform and your connectors. If you are a subscriber, please contact Confluent Support at firstname.lastname@example.org for more information.
See Confluent Platform license for license properties and License topic configuration for information about the license topic.
For a complete list of configuration properties for this connector, see Confluent Replicator Configuration Properties.
For an example of how to get Kafka Connect connected to Confluent Cloud, see Distributed Cluster in Connect Kafka Connect to Confluent Cloud.
See Tutorial: Replicating Data Between Clusters in Multi-DC Deployment Architectures.