Replicators¶
Confluent Replicator replicates Kafka topics from one cluster to another (source to destination), copying the messages, and creating replica topics that preserve the topic configuration in the source cluster.
From the Replicators pages in Control Center, you can:
- Monitor tasks, message throughput, and Connect workers running as replicators.
- Monitor metrics on source topics on the origin cluster.
- Monitor metrics on replicated topics on the destination cluster.
- Drill down on source and replicated topics to monitor and configure them through the Control Center Topics pages (see Manage Topics).
The metrics provided are:
- Throughput - the number of messages replicated per second.
- Message Lag - the number of messages that have been produced to the origin cluster that have not yet been replicated to the destination.
- Latency - the average time period between message production to the origin cluster and message production to the destination cluster.
The metrics are broken down by connector, task and topic/partition.
Enable Replicator monitoring¶
If you are just getting started with Replicator, you can use the Tutorial: Replicating Data Across Clusters as a guide to bring up origin and destination clusters with topic replication, then follow the steps in the last section, Use Control Center to monitor replicators, to configure monitoring.
All Replicators page¶
To access the Replicators overview page, select a cluster from the navigation bar and click Replicators from the cluster submenu.
The Replicators overview page lists all replicators in the selected cluster, and shows status at a glance of tasks for each replicator, including:
- Status of replicators (running, failed, and paused)
- Number of source topics processed
- Number of messages replicated per second
- Message lag
- Latency
Origin and Destination clusters¶
If your clusters and replication are configured properly, a minimum of two clusters, for example origin and destination, will be displayed in the Environment overview.
Tip
If you do not see fully populated origin and destination clusters on Control Center, review the configuration details in Tutorial: Replicating Data Across Clusters and particularly the subtopic in that guide, Use Control Center to monitor replicators.
The clusters are rendered on Control Center with auto-generated names, based on your configuration.
Optionally, you can edit the cluster names:
- Select a cluster.
- Click Cluster settings on the navigation menu.
- On the General tab, type a cluster name, and click Save.
Here, you can also configure other cluster wide settings such as broker defaults, Self-Balancing, and storage.
In the associated example from the Replicator Tutorial, Replicator is set up to replicate topic data from origin to destination.
The destination cluster settings are shown here.
The origin cluster settings are shown here.
Source and Replicated topics¶
Source topics are located on the origin cluster. There are several ways to navigate to these topics. For example:
- With either cluster selected, click Replicators on the navigation menu, click a replicator, then choose Source Topics tab -> <topic>.
- With either cluster selected, click Replicators on the navigation menu, click a replicator, then click the Throughput metrics card -> <task> -> <topic>.
- Select the origin cluster, click Topics, and drill down on a topic that way.
Tip
Not all topics on the origin cluster will necessarily be replicated;
only topics configured for replication
will be replicated. Also, some internal topics live on the origin cluster
and are not replicated, such as __consumer_timestamps which the Replicator
source consumer reads from to perform consumer offset translation.
Replicated topics live on the destination cluster. To navigate to these topics:
Select the destination cluster.
Select Topics from the navigation menu.
Click a replicated topic.
The following sections describe the available replication monitoring, metrics, and configuration views in more detail.
Metrics and topics for a replicator¶
To view details on a specific replicator, click a replicator in the list on the All Replicators page.
The Replicator status page is displayed, showing:
- Throughput metrics on message replication, including messages replicated per second, number of source topics, lag, and latency
- Task overview, including total tasks, and number of running, failed, and paused tasks
- List of Connect workers showing status of each worker, along with message performance metrics per worker. You can scroll and search the list of workers.
Tasks associated with a replicator¶
To view a list of tasks associated with a replicator, click the Throughput card for the selected replicator.
The list of tasks is displayed.
Message metrics for a task¶
To view metrics for a specific task, click one of the tasks associated with the selected replicator.
The task metrics are displayed at the top of the page. A list of source topics and associated metrics per topic is displayed on the lower half of the page. You can search and scroll through this list.
Source topics¶
To view details on a source topic (a topic on the origin cluster), select the Source Topics tab for the Replicator, then click a topic in the list.
You can also view details on source topic by clicking a topic on the task metrics page or, with the origin cluster selected, click Topics from the navigation bar, then click a topic in that list.
An overview of metrics for the source topic is displayed.
You have the same monitoring and configuration options on both source and replicated topics.
Replicated topics¶
To view details on a replicated topic (on the destination cluster), select the Destination cluster, then click Topics on the Control Center menu to show all topics in the cluster.
Click one of the replicas in the topic list to bring up the overview page for that topic.
Click the metrics cards on the Overview tab to drill down on the monitoring data for that topic.
For example, click into Production to show producer metrics.
Click Consumption to show consumer metrics.
Navigate using the other tabs to view and manage details on the topic. For example, this page shows Messages associated with the replicated topic.
This page shows the topic configuration and provides options to re-configure it. (Click Show full config to get to advanced configuration options.)
Click Edit settings to modify basic configurations on the topic. Click Switch to expert mode for more options.
You have the same monitoring and configuration options on both source and replicated topics, and can configure these independently. For example, you might choose to reduce the retention time or size on replicated topic but keep these the same on the source topic.
To learn more about working with topics in Control Center, see Manage Topics.