Replicators in Control Center

Confluent Replicator replicates Kafka topics from one cluster to another (source to destination), copying the messages, and creating replica topics that preserve the topic configuration in the source cluster.

From the Replicators pages in Control Center, you can:

  • Monitor tasks, message throughput, and Connect workers running as replicators.
  • Monitor metrics on source topics on the origin cluster.
  • Monitor metrics on replicated topics on the destination cluster.
  • Drill down on source and replicated topics to monitor and configure them through the Control Center Topics pages (see Manage Topics in Control Center).

The metrics provided are:

  • Throughput - the number of messages replicated per second.
  • Message Lag - the number of messages that have been produced to the origin cluster that have not yet been replicated to the destination.
  • Latency - the average time period between message production to the origin cluster and message production to the destination cluster.

The metrics are broken down by connector, task and topic/partition.

Enable Replicator monitoring

If you are just getting started with Replicator, you can use the Tutorial: Replicate Data Across Kafka Clusters in Confluent Platform as a guide to bring up origin and destination clusters with topic replication, then follow the steps in the last section, Use Control Center to monitor replicators, to configure monitoring.

All Replicators page

To access the Replicators overview page, select a cluster from the navigation bar and click Replicators from the cluster submenu.

../_images/c3-replicators-all.png

The Replicators overview page lists all replicators in the selected cluster, and shows status at a glance of tasks for each replicator, including:

  • Status of replicators (running, failed, and paused)
  • Number of source topics processed
  • Number of messages replicated per second
  • Message lag
  • Latency

Origin and Destination clusters

If your clusters and replication are configured properly, a minimum of two clusters, for example origin and destination, will be displayed in the Environment overview.

../_images/c3-replicators-multi-cluster.png

Tip

If you do not see fully populated origin and destination clusters on Control Center, review the configuration details in Tutorial: Replicate Data Across Kafka Clusters in Confluent Platform and particularly the subtopic in that guide, Use Control Center to monitor replicators.

The clusters are rendered on Control Center with auto-generated names, based on your configuration.

Optionally, you can edit the cluster names:

  1. Select a cluster.
  2. Click Cluster settings on the navigation menu.
  3. On the General tab, type a cluster name, and click Save.

Here, you can also configure other cluster wide settings such as broker defaults, Self-Balancing, and storage.

In the associated example from the Replicator Tutorial, Replicator is set up to replicate topic data from origin to destination.

The destination cluster settings are shown here.

../_images/c3-replicators-cluster-name.png

The origin cluster settings are shown here.

../_images/c3-replicators-cluster-name-origin.png

Source and Replicated topics

Source topics are located on the origin cluster. There are several ways to navigate to these topics. For example:

  • With either cluster selected, click Replicators on the navigation menu, click a replicator, then choose Source Topics tab -> <topic>.
  • With either cluster selected, click Replicators on the navigation menu, click a replicator, then click the Throughput metrics card -> <task> -> <topic>.
  • Select the origin cluster, click Topics, and drill down on a topic that way.

Tip

Not all topics on the origin cluster will necessarily be replicated; only topics configured for replication will be replicated. Also, some internal topics live on the origin cluster and are not replicated, such as __consumer_timestamps which the Replicator source consumer reads from to perform consumer offset translation.

Replicated topics live on the destination cluster. To navigate to these topics:

  1. Select the destination cluster.

  2. Select Topics from the navigation menu.

    ../_images/c3-replicator-topics-overview.png

  3. Click a replicated topic.

    ../_images/c3-replicated-topic-drilldown-overview.png

The following sections describe the available replication monitoring, metrics, and configuration views in more detail.

Metrics and topics for a replicator

To view details on a specific replicator, click a replicator in the list on the All Replicators page.

../_images/c3-replicators-drilldown.png

The Replicator status page is displayed, showing:

  • Throughput metrics on message replication, including messages replicated per second, number of source topics, lag, and latency
  • Task overview, including total tasks, and number of running, failed, and paused tasks
  • List of Connect workers showing status of each worker, along with message performance metrics per worker. You can scroll and search the list of workers.
../_images/c3-replicators-status.png

Tasks associated with a replicator

To view a list of tasks associated with a replicator, click the Throughput card for the selected replicator.

../_images/c3-replicators-throughput.png

The list of tasks is displayed.

../_images/c3-replicators-metrics.png

Message metrics for a task

To view metrics for a specific task, click one of the tasks associated with the selected replicator.

../_images/c3-replicators-task-drilldown.png

The task metrics are displayed at the top of the page. A list of source topics and associated metrics per topic is displayed on the lower half of the page. You can search and scroll through this list.

../_images/c3-replicators-task-metrics-and-source-topics.png

Source topics

To view details on a source topic (a topic on the origin cluster), select the Source Topics tab for the Replicator, then click a topic in the list.

../_images/c3-replicators-source-topics-select.png

You can also view details on source topic by clicking a topic on the task metrics page or, with the origin cluster selected, click Topics from the navigation bar, then click a topic in that list.

An overview of metrics for the source topic is displayed.

../_images/c3-replicators-source-topic-drilldown-overview.png

You have the same monitoring and configuration options on both source and replicated topics.

Replicated topics

To view details on a replicated topic (on the destination cluster), select the Destination cluster, then click Topics on the Control Center menu to show all topics in the cluster.

../_images/c3-replicator-topics-overview.png

Click one of the replicas in the topic list to bring up the overview page for that topic.

../_images/c3-replicated-topic-drilldown-overview.png

Click the metrics cards on the Overview tab to drill down on the monitoring data for that topic.

For example, click into Production to show producer metrics.

../_images/c3-replicated-topic-metrics-production.png

Click Consumption to show consumer metrics.

../_images/c3-replicated-topic-metrics-consumer-lag.png

Navigate using the other tabs to view and manage details on the topic. For example, this page shows Messages associated with the replicated topic.

../_images/c3-replicator-topic-drilldown-messages.png

This page shows the topic configuration and provides options to re-configure it. (Click Show full config to get to advanced configuration options.)

../_images/c3-replicator-topic-drilldown-config.png

Click Edit settings to modify basic configurations on the topic. Click Switch to expert mode for more options.

../_images/c3-replicator-topic-drilldown-config-edit.png

You have the same monitoring and configuration options on both source and replicated topics, and can configure these independently. For example, you might choose to reduce the retention time or size on replicated topic but keep these the same on the source topic.

To learn more about working with topics in Control Center, see Manage Topics in Control Center.