Confluent Replicator replicates Kafka topics from one cluster to another (source to
destination), copying the messages, and creating replica topics that preserve the
topic configuration in the source cluster.
The metrics are broken down by connector, task and topic/partition.
All Replicators page
To access the Replicators overview page, select a cluster from the
navigation bar and click Replicators from the cluster submenu.
The Replicators overview page lists all replicators in the selected cluster, and shows status at a glance of
tasks for each replicator, including:
- Status of replicators (running, failed, and paused)
- Number of source topics processed
- Number of messages replicated per second
- Message lag
Origin and Destination clusters
If your clusters and replication are configured properly, a minimum of two clusters, for example origin and destination, will be displayed in the Environment overview.
The clusters are rendered on Control Center with auto-generated names, based on your configuration.
Optionally, you can edit the cluster names:
- Select a cluster.
- Click Cluster settings on the navigation menu.
- On the General tab, type a cluster name, and click Save.
Here, you can also configure other cluster wide settings such as broker defaults, Self-Balancing, and storage.
In the associated example from the Replicator Tutorial, Replicator is set up to replicate topic data from origin to destination.
The destination cluster settings are shown here.
The origin cluster settings are shown here.
Source and Replicated topics
Source topics are located on the origin cluster. There are several ways to navigate to these topics. For example:
- With either cluster selected, click Replicators on the navigation menu, click a replicator, then choose Source Topics tab -> <topic>.
- With either cluster selected, click Replicators on the navigation menu, click a replicator, then click the Throughput metrics card -> <task> -> <topic>.
- Select the origin cluster, click Topics, and drill down on a topic that way.
Replicated topics live on the destination cluster. To navigate to these topics:
Select the destination cluster.
Select Topics from the navigation menu.
Click a replicated topic.
The following sections describe the available replication monitoring, metrics, and configuration views in more detail.
Metrics and topics for a replicator
To view details on a specific replicator, click a replicator in the list on the All Replicators page.
The Replicator status page is displayed, showing:
- Throughput metrics on message replication, including messages replicated per second, number of source topics, lag, and latency
- Task overview, including total tasks, and number of running, failed, and paused tasks
- List of Connect workers showing status of each worker, along with message performance metrics per worker. You can scroll and search the list of workers.
Tasks associated with a replicator
To view a list of tasks associated with a replicator, click the Throughput card for the selected replicator.
The list of tasks is displayed.
Message metrics for a task
To view metrics for a specific task, click one of the tasks associated with the selected replicator.
The task metrics are displayed at the top of the page. A list of source topics and associated metrics per topic is displayed on the lower half of the page. You can search and scroll through this list.
To view details on a source topic (a topic on the origin cluster), select the Source Topics tab for the Replicator, then click a topic in the list.
You can also view details on source topic by clicking a topic on the task metrics page or, with the origin cluster selected, click Topics from the navigation bar, then click a topic in that list.
An overview of metrics for the source topic is displayed.
You have the same monitoring and configuration options on both source and replicated topics.
To view details on a replicated topic (on the destination cluster), select the Destination cluster, then click Topics on the Control Center menu to show all topics in the cluster.
Click one of the replicas in the topic list to bring up the overview page for that topic.
Click the metrics cards on the Overview tab to drill down on the monitoring data for that topic.
For example, click into Production to show producer metrics.
Click Consumption to show consumer metrics.
Navigate using the other tabs to view and manage details on the topic. For example, this page shows Messages associated with the replicated topic.
This page shows the topic configuration and provides options to re-configure it. (Click Show full config to get to advanced configuration options.)
Click Edit settings to modify basic configurations on the topic. Click Switch to expert mode for more options.
You have the same monitoring and configuration options on both source and replicated topics, and can
configure these independently. For example, you might choose to reduce the retention time or size on
replicated topic but keep these the same on the source topic.
To learn more about working with topics in Control Center, see Topics.