documentation
Get Started Free
  • Get Started Free
  • Stream
      Confluent Cloud

      Fully-managed data streaming platform with a cloud-native Kafka engine (KORA) for elastic scaling, with enterprise security, stream processing, governance.

      Confluent Platform

      An on-premises enterprise-grade distribution of Apache Kafka with enterprise security, stream processing, governance.

  • Connect
      Managed

      Use fully-managed connectors with Confluent Cloud to connect to data sources and sinks.

      Self-Managed

      Use self-managed connectors with Confluent Platform to connect to data sources and sinks.

  • Govern
      Managed

      Use fully-managed Schema Registry and Stream Governance with Confluent Cloud.

      Self-Managed

      Use self-managed Schema Registry and Stream Governance with Confluent Platform.

  • Process
      Managed

      Use Flink on Confluent Cloud to run complex, stateful, low-latency streaming applications.

      Self-Managed

      Use Flink on Confluent Platform to run complex, stateful, low-latency streaming applications.

Stream
Confluent Cloud

Fully-managed data streaming platform with a cloud-native Kafka engine (KORA) for elastic scaling, with enterprise security, stream processing, governance.

Confluent Platform

An on-premises enterprise-grade distribution of Apache Kafka with enterprise security, stream processing, governance.

Connect
Managed

Use fully-managed connectors with Confluent Cloud to connect to data sources and sinks.

Self-Managed

Use self-managed connectors with Confluent Platform to connect to data sources and sinks.

Govern
Managed

Use fully-managed Schema Registry and Stream Governance with Confluent Cloud.

Self-Managed

Use self-managed Schema Registry and Stream Governance with Confluent Platform.

Process
Managed

Use Flink on Confluent Cloud to run complex, stateful, low-latency streaming applications.

Self-Managed

Use Flink on Confluent Platform to run complex, stateful, low-latency streaming applications.

Learn
Get Started Free
  1. Home
  2. Platform
  3. Multi-DC Deployment Architectures
  4. Cluster Linking
  5. Cluster Linking

CONFLUENT PLATFORM

  • Overview
  • Get Started
    • What is Confluent Platform?
    • Quick Start for Confluent Platform
    • Kafka Basics on Confluent Platform
    • Apache Kafka Introduction
    • Videos, Demos, and Reading Material
      • Scripted Confluent Platform Demo
        • Overview
        • Deploy Confluent Platform Environment
        • Deploy Hybrid Confluent Platform and Cloud Environment
        • End Demo
        • Troubleshooting
      • Tutorial: Introduction to Streaming Application Development
      • Clickstream Data Analysis Pipeline Using ksqlDB
      • RBAC Example for Confluent Platform
      • Replicator Schema Translation Example for Confluent Platform
      • DevOps for Kafka with Kubernetes and GitOps
        • Overview
        • Kafka DevOps Case Studies
          • Case Study: Graduated Environments
          • Case Study: Manage Cloud Secrets
          • Case Study: Kafka Connect management with GitOps
    • Resources
  • Install and Upgrade
    • Overview
    • System Requirements
    • Install Manually
      • ZIP and TAR
      • Ubuntu and Debian
      • RHEL and CentOS
      • Docker
        • Install using Docker
        • Docker Configuration Parameters
        • Docker Image Reference
        • Docker Security
        • Docker Developer Guide
      • Configure Automatic Startup and Monitoring
    • Deploy with Ansible Playbooks
    • Deploy with Confluent for Kubernetes
    • License
    • Upgrade Confluent Platform
      • Overview
      • Upgrade Confluent Platform
    • Supported Versions and Interoperability
    • Installation Packages
    • Migrate from Apache Kafka
    • Migrate an Existing Kafka Deployment
    • Migrate to Confluent Server
    • Migrate from ZooKeeper to KRaft (EA)
  • Build Client Applications
    • Overview
    • Configure Clients
      • Consumer
      • Producer
      • Schemas, Serializers, and Deserializers
      • Configuration Properties
    • Client Guides
      • Python
      • .NET Client
      • JavaScript Client
      • Go Client
      • C++ Client
      • Java Client
      • JMS Client
        • Overview
        • Development Guide
    • Client Examples
      • Overview
      • Python Client
      • .NET Client
      • JavaScript Client
      • Go Client
      • C++ Client
      • Java
      • Spring Boot
      • KafkaProducer
      • REST
      • Clojure
      • Groovy
      • Kafka Connect Datagen
      • kafkacat
      • Kotlin
      • Ruby
      • Rust
      • Scala
    • Kafka Client APIs
      • Python Client API
      • .NET Client API
      • JavaScript Client API
      • Go Client API
      • C++ Client API
      • Java Client API
      • JMS Client
        • Overview
        • Development Guide
    • Deprecated Client APIs
    • MQTT Proxy
      • Introduction
      • Communication Security Settings
      • MQTT Proxy Configuration Options
    • kcat (formerly kafkacat) Utility
  • Confluent REST Proxy
    • Overview
    • Quick Start
    • Rest Proxy API Reference
    • Production Deployment
      • Confluent Server Admin APIs
        • Configuration
        • Security
      • Standalone REST Proxy
        • Overview
        • Configuration
        • Monitoring
        • Security
    • REST Proxy Tutorial
    • Connect to Confluent Cloud
  • ksqlDB and Kafka Streams
    • Overview
    • ksqlDB
      • ksqlDB Overview
      • ksqlDB Quickstart
      • Install ksqlDB
      • Operate ksqlDB
      • Upgrade ksqlDB
      • Develop applications for ksqlDB
      • Run ksqlDB in Confluent Cloud
      • Connect ksqlDB to Confluent Cloud
      • Migrate Confluent Cloud ksqlDB applications
      • Run ksqlDB in Confluent Control Center
      • Connect ksqlDB to Confluent Control Center
      • Secure ksqlDB with RBAC
      • Frequently Asked Questions
      • Troubleshoot ksqlDB issues
      • Tutorials and Examples
        • Tutorials overview
        • ksqlDB Quick Start
        • How-to Guides
        • Example Code Snippets
        • Materialized View/cache
        • Streaming ETL Pipeline
        • Event-driven Microservice
        • ksqlDB with Embedded Connect
        • Clickstream Data Analysis Pipeline Using ksqlDB
        • Kafka Tutorials Using ksqlDB
    • Kafka Streams
      • Kafka Streams Overview
      • Introduction
      • Build your first Streams application
      • Tutorial: Introduction to Streaming Application Development
      • Connect Confluent Platform Components to Confluent Cloud
      • Streams Concepts
      • Streams Architecture
      • Streams Code Examples
      • Streams Developer Guide
        • Writing a Streams Application
        • Testing Streams Code
        • Configuring a Streams Application
        • Streams DSL
        • Naming Kafka Streams DSL Topologies
        • Optimizing Kafka Streams Topologies
        • Processor API
        • Data Types and Serialization
        • Interactive Queries
        • Memory Management
        • Running Streams Applications
        • Managing Streams Application Topics
        • Kafka Streams Security
        • Application Reset Tool
      • Pipelining with Kafka Connect and Kafka Streams
      • Streams Operations
        • Capacity Planning and Sizing
        • Monitoring Kafka Streams Applications
      • Streams Upgrade Guide
      • Streams FAQ
      • Streams Javadocs
  • Connect to External Systems
    • Overview
    • Get Started
    • Kafka Connect 101
    • Connectors
    • Confluent Hub
      • Overview
      • Confluent Hub Client
      • Command Reference
        • Overview
        • confluent-hub help
        • confluent-hub install
      • Component Archive Specification
      • Contribute to Confluent Hub
    • Connect on z/OS
    • Install
    • License
    • Supported
    • Preview
    • Configure
    • Monitor
    • Logging
    • Connect to Confluent Cloud
    • Developer Guide
    • Tutorial: Moving Data In and Out of Kafka
    • Reference
      • Connect Javadocs
      • Connect REST Interface
      • Worker Configuration Properties
      • Connector Configuration Properties
    • Transform
    • Security
      • Kafka Connect Security Basics
      • Kafka Connect and RBAC
        • Get Started With RBAC and Kafka Connect
        • Configure RBAC for a Connect Cluster
        • Configure RBAC for a Connect Worker
        • RBAC for self-managed connectors
        • Connect Secret Registry
        • Example Connect role-binding sequence
    • Design
    • Add Connectors and Software
    • Install Community Connectors
    • Upgrade
    • FileStream Connectors
    • FAQ
  • Schema Management
    • Overview
    • Tutorial
    • Installing and Configuring
      • Install
      • Configure Schema Registry
      • Configure Clients to Schema Registry
      • Deploy in Production
      • Deployment Architectures
      • Migrate Schemas
    • Fundamentals
      • Key Concepts
      • Schema Evolution and Compatibility for Schema Registry on Confluent Platform
      • Schema Formats for Schema Registry on Confluent Platform
        • Serializers and Deserializers Overview
        • Avro
        • Protobuf
        • JSON Schema
      • Data Contracts
    • Manage Schemas
      • Manage Schemas in Control Center
      • Schema Contexts
      • Schema Linking
      • Schema Validation
      • Monitoring
      • Delete Schemas
      • Integrate Schemas from Connectors
    • Security
      • Schema Registry Security Overview
      • Role-Based Access Control
      • Schema Registry Security Plugin
        • Overview
        • Install and Configure
        • Schema Registry Authorization
          • Supported Operations and Resources
          • Role-Based Access Control
          • Schema Registry ACL Authorizer
          • Topic ACL Authorizer
    • Reference
      • Overview
      • Maven Plugin
      • API Reference
      • API Usage Examples
    • FAQ
  • Security
    • General Security
      • Security Overview
      • Security Tutorial
      • Configuring Confluent Server Authorizer
      • Cluster Registry
      • Security Compliance
      • Prefixes for Configuring Security
    • Authentication
      • Authentication Methods Overview
      • Authentication with SASL
        • Authentication with SASL using JAAS
        • Install
        • Configuring GSSAPI
        • Configuring OAUTHBEARER
        • Configuring PLAIN
        • Configuring SCRAM
        • Authentication using Delegation Tokens
        • Configuring Kafka Client Authentication with LDAP
      • Encrypt and Authenticate with TLS
      • HTTP Basic Authentication
      • Adding security to a running cluster
    • Authorization
      • Authorization using Role-Based Access Control
        • RBAC Overview
        • Quick Start
        • Predefined Roles
        • Enable RBAC in a Running Cluster
        • Discover Identifiers for Clusters
        • Configuring Token Authentication
        • Confluent Metadata API Reference
        • RBAC Example for Confluent Platform
      • Configure RBAC using the REST API
      • ACLs
        • Authorization using centralized ACLs
        • Authorization using Access Control Lists (ACLs)
      • Group-Based Authorization Using LDAP
        • Configuring Confluent Server Authorizer
        • Configuring LDAP
        • Tutorial: Group-Based Authorization Using LDAP
    • Data Protection
      • Audit Logs
        • Audit Log Concepts
        • Auditable Events
        • Configure Audit Logs using the Confluent CLI
        • Configure MDS to Manage Centralized Audit Logs
        • MDS API Audit Log Configuration
        • Configure Audit Logs using the Properties File
      • Encrypt with TLS
      • Secrets
        • Secrets Management
        • Tutorial: Secret Protection
      • Redact Confluent Logs
    • Component Security
      • Confluent Control Center (Legacy) Security
        • Overview
        • Configure TLS/SSL
        • Configure SASL
        • Configure HTTP Basic Authentication
        • Authorize with Kafka ACLs
        • Configure LDAP
        • Configure RBAC
        • Manage and View RBAC Roles
          • Log in to Control Center (Legacy) when RBAC enabled
          • Manage RBAC roles with Control Center (Legacy)
          • View your RBAC roles in Control Center (Legacy)
      • Kafka Streams Security
      • Schema Registry Security
      • Kafka Connect Security
        • Kafka Connect Security Basics
        • Kafka Connect and RBAC
          • Get Started With RBAC and Kafka Connect
          • Configure RBAC for a Connect Cluster
          • Configure RBAC for a Connect Worker
          • RBAC for self-managed connectors
          • Connect Secret Registry
          • Example Connect role-binding sequence
      • ksqlDB RBAC
      • REST Proxy Security
        • REST Proxy Security
        • REST Proxy Security Plugins
      • ZooKeeper Security
    • Security Management Tools
      • Ansible Playbooks for Confluent Platform
      • Docker Security for Confluent Platform
  • Multi-DC Deployment Architectures
    • Overview
    • Multi-Data Center Architectures
    • Cluster Linking
      • Cluster Linking Overview
      • Use Cases and Tutorials
        • Share Data Across Topics
        • Create Hybrid Cloud and Bridge-to-Cloud Deployments
        • Migrate Data
      • Configure, Manage, and Monitor
        • Manage Mirror Topics
        • Configure and Manage
        • Commands Reference
        • Monitor Metrics
        • Manage Security
      • FAQ
    • Multi-Region Clusters
      • Overview
      • Tutorial: Multi-Region Clusters
      • Transition Standard Active-Passive Data Centers to a Multi-Region Stretched Cluster
    • Replicator for Multi-Datacenter Replication
      • Replicator Overview
      • Tutorial: Replicating Data Across Clusters
      • Active-active Demo
      • Download and Install
      • Configure and Run
      • Tuning Replicator
      • Monitoring Replicator
      • Replicator and Cross-Cluster Failover
      • Configuration Options
      • Migrate from MirrorMaker to Replicator
      • Replicator Verifier
      • Replicator to Cloud Configurations
  • Configure and Manage
    • Confluent Control Center (Legacy)
      • Control Center Overview
      • Installing and Configuring Control Center (Legacy)
        • Configuring
        • Configuration Reference
        • Check Control Center (Legacy) Version and Enable Auto-Update
        • Properties File
        • Connecting Control Center to Confluent Cloud
        • Confluent Monitoring Interceptors in Control Center (Legacy)
        • Installing Control Center (Legacy) on Kafka
        • Managing Confluent Platform Licenses
        • Troubleshooting Control Center (Legacy)
        • Upgrading Control Center (Legacy)
      • Clusters
      • Brokers
      • Topics Overview
        • Topics Overview
        • Create
        • View Topic Metrics
        • Message Browser
        • Manage Schemas for Topics
        • Edit Configuration Settings for Topics
        • Delete Topics
      • Connect
      • ksqlDB
      • Consumers
      • Replicators
      • Clusters
      • Control Center (Legacy) Alerts
        • Overview
        • Access Alerts and Alert History
        • Configure Alerts Properties
        • View and Manage Triggers
        • Manage Actions
        • Configure PagerDuty Emails with Alerts
        • REST API for Alerts History
        • Example Triggers and Actions
        • Troubleshooting Alerts
      • Security
    • Configuration Reference
      • Overview
      • Broker and Controller Configuration
      • Topic Configuration
      • Consumer Configuration
      • Producer Configuration
      • Connect Configuration
        • Overview
        • Sink Configurations
        • Source Configurations
      • Admin Client Configuration
      • License Configuration
      • Kafka Streams Configuration Reference
      • Kafka Streams Configuration Guide
      • Docker Configuration Parameters for Confluent Platform
      • Control Center Configurations
      • ZooKeeper Configurations
      • KRaft Configurations
      • ksqlDB Server Configurations
    • Change Configuration Settings Dynamically
    • CLI Tools for Confluent Platform
      • CLI Tools for Confluent Platform
      • Confluent CLI
    • Kafka Metadata Management
      • Metadata Management in Kafka
      • KRaft Overview
      • Configure KRaft
      • Configure ZooKeeper for Production
    • Configure a Multi-Node Environment with Docker
    • Confluent Platform Metadata Service (MDS)
      • Configure the Confluent Platform Metadata Service (MDS)
      • Configure Confluent Platform Components to Communicate with MDS over TLS/SSL
      • Configure mTLS Authentication and RBAC for Kafka Brokers
      • Configure Kerberos Authentication for Brokers Running MDS
      • Configure LDAP Authentication
      • Configure LDAP Group-Based Authorization for MDS
      • Configure MDS to Manage Centralized Audit Logs
      • Metadata Service Configuration Settings
      • Confluent Metadata API Reference
    • Confluent Metrics Reporter
    • Confluent Health+
      • Health+ Overview
      • Enable Health+
      • Health+ Intelligent Alerts
      • Health+ Monitoring Dashboard
      • Confluent Telemetry Reporter
      • Telemetry Reporter Metrics
      • Confluent Health+ FAQ
    • Confluent REST Proxy
      • Overview
      • Quick Start
      • Rest Proxy API Reference
      • Production Deployment
        • Confluent Server Admin APIs
          • Configuration
          • Security
        • Standalone REST Proxy
          • Overview
          • Configuration
          • Monitoring
          • Security
      • REST Proxy Tutorial
      • Connect to Confluent Cloud
    • Kafka Operations
      • Overview
      • Running Kafka in Production
      • Post Kafka Deployment
      • Self-Balancing Clusters
        • Self-Balancing Overview
        • Quick Start Demo (Docker)
        • Tutorial: Adding and Removing Brokers
        • Configuration Options and Commands
        • Performance and Resource Usage
      • Auto Data Balancing
        • Quick Start
        • Tutorial (Docker)
        • Command and Configuration Options
      • Monitoring Kafka with JMX
      • Confluent Metrics Reporter
      • Tiered Storage
    • Docker Operations
      • Overview
      • Kafka Monitoring and Metrics Using JMX
      • Configure Docker Logging
      • Mounting Docker External Volumes
    • Post Kafka Deployment
    • Kafka Streams Operations
      • Capacity Planning and Sizing
      • Monitoring Kafka Streams Applications
    • ksqlDB Operations
    • DevOps for Kafka with Kubernetes and GitOps
      • Overview
      • Kafka DevOps Case Studies
        • Case Study: Graduated Environments
        • Case Study: Manage Cloud Secrets
        • Case Study: Kafka Connect management with GitOps
  • Confluent Health+
    • Health+ Overview
    • Enable Health+
    • Health+ Intelligent Alerts
    • Health+ Monitoring Dashboard
    • Confluent Telemetry Reporter
    • Telemetry Reporter Metrics
    • Confluent Health+ FAQ
  • Confluent CLI
  • Release Notes
    • Release Notes
    • Component Changelogs
  • APIs and Javadocs
    • Overview
    • Kafka API Javadocs
      • Kafka Java Client APIs
      • Kafka Producer Java API
      • Kafka Consumer Java API
      • Kafka AdminClient Java API
      • Kafka Common Java API
      • Kafka Streams Java API
      • Kafka Connect Java API
    • Client APIs
      • Python Client API
      • .NET Client API
      • JavaScript Client API
      • Go Client API
      • C++ Client API
      • Java Client API
      • JMS Client
        • Overview
        • Development Guide
    • Confluent APIs
      • Confluent REST Proxy API
        • Overview
        • Quick Start
        • Rest Proxy API Reference
        • Production Deployment
          • Confluent Server Admin APIs
            • Configuration
            • Security
          • Standalone REST Proxy
            • Overview
            • Configuration
            • Monitoring
            • Security
        • REST Proxy Tutorial
        • Connect to Confluent Cloud
      • Connect REST API
      • ksqlDB REST API
      • Metadata API
      • Schema Registry API
  • Glossary

Monitoring Cluster Metrics and Optimizing Links for Cluster Linking on Confluent Platform¶

Confluent Platform exposes several metrics through Java Management Extensions (JMX) that are useful for monitoring Cluster Linking. Some of these are derived from extending existing interfaces, and others are new for Cluster Linking.

To monitor Cluster Linking, set the JMX_PORT environment variable before starting the cluster, then collect the reported metrics using your usual monitoring tools. JMXTrans, Graphite, and Grafana are a popular combination for collecting and reporting JMX metrics from Kafka. Datadog is another popular monitoring solution.

Wherever possible, metrics are categorized below per associated link names.

Also, you can set quotas to limit resources and bandwidth used by Cluster Linking to help optimize the system.

Cluster Link Fetcher Metrics¶

kafka.server.link:type=ClusterLinkFetcherManager,name=MaxLag,clientId=ClusterLink,link-name={linkName}
Maximum lag in messages between the replicas on the destination cluster and the leader replica on the source cluster. Provides an indication of overall cluster link health and a relative measure of whether the destination cluster is lagging behind.
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=ClusterLinkFetcherThread-{destBrokerId}-{linkName}-{sourceBrokerId},topic={topic},partition={partition},link-name={linkName}
Drill-down of MaxLag, which provides lag in number of messages per replica on the destination cluster from the leader replica on the source cluster. Useful to debug any specific lagging topic partitions/replicas on the destination cluster.
kafka.server.link:type=ClusterLinkFetcherManager,name=DeadThreadCount,clientId=ClusterLink,link-name={linkName}
Number of dead threads of type ClusterLinkThread, where a dead thread is indicative of an issue with fetching from the source cluster for a partition.
kafka.server:type=FetcherStats,name=BytesPerSec,clientId=ClusterLinkFetcherThread-{destBrokerId}-{linkName}-{sourceBrokerId},brokerHost={host},brokerPort={port},link-name={linkName}
Rate at which data is fetched from the source cluster. Indicates amount of throughput in bytes per second on the cluster link.
kafka.server:type=FetcherStats,name=RequestsPerSec,clientId=ClusterLinkFetcherThread-{destBrokerId}-{linkName}-{sourceBrokerId}, brokerHost={host},brokerPort={port},link-name={linkName}
Rate at which requests are issued to the source cluster over the cluster link.

Throttle Time Metrics for Cluster Link Fetchers¶

kafka.server:type=cluster-link,link-name={linkName}
Throttle time metrics may indicate that a cluster link connection is being throttled, which is useful for understanding why lag may be increasing over a cluster link.
Name Description
fetch-throttle-time-avg, fetch-throttle-time-max Gives throttle times for the cluster linking fetchers. May indicate increases in lag on the cluster link due to throttling/quotas being enforced.

Cluster Link Network Client Metrics¶

kafka.server:type=cluster-link-metadata-metrics,link-name={linkName}
Metrics pertaining to the metadata refresh client in the cluster link.
kafka.server:type=cluster-link-fetcher-metrics,link-name={linkName},broker-id={id},fetcher-id={id}
Metrics pertaining to data fetcher requests in the cluster link.

Metrics seen through cluster-link-fetcher-metrics are shown in the following table.

Name Description
connection-count Number of connections for a cluster link
connection-creation-rate, connection-creation-total Rate per second and total for connections created for the cluster link. If the rate is high, the source cluster may be overloaded with connections.
connection-close-rate, connection-close-total Rate per second and total for connections closed for the cluster link. Can be compared to connection creation to understand the balance of creating/closing connections.
incoming-byte-rate, incoming-byte-total Number of bytes per second and total received from the source cluster.
outgoing-byte-rate, outgoing-byte-total Number of bytes per second and total sent to the source cluster.
network-io-rate, network-io-total Indicates rate and total network input/output (IO).
request-rate, request-total Rate per second and total requests issued to the source cluster over the cluster link.
response-rate, response-total Rate per second and total number of responses received from the source cluster over the cluster link.
request-size-avg, request-size-max Average and maximum size in bytes of requests issued to the source cluster over the cluster link.
io-ratio, io-wait-time-ns-avg, io-waittime-total, iotime-total Statistics of the destination cluster’s network IO for the requests over the cluster link.
successful-authentication-rate, successful-authentication-total Rate per second and total number of cluster link clients authenticating to the source cluster over the cluster link.
successful-reauthentication-rate, successful-reauthentication-total Rate per second and total re-authentication to the source cluster over the cluster link.
failed-reauthentication-rate, failed-reauthentication-total Rate per second and total failed re-authentications to the source cluster over the cluster link. If failures are present, it could indicate misconfigured or stale cluster link credentials.
reauthentication-latency-avg Average re-authentication latency. Helps to assess whether clients are taking too long to authenticate to the clusters.

Request Metrics for Cluster Link APIs¶

kafka.network:type=RequestMetrics,name={ LocalTimeMs| RemoteTimeMs | RequestBytes | RequestQueueTimeMs| ResponseQueueTimeMs | ResponseSendIoTimeMs | ResponseSendTimeMs | ThrottleTimeMs| TotalTimeMs },request={CreateClusterLinks| DeleteClusterLinks| ListClusterLinks}
Depending on the request name, provides statistics on requests on the cluster link, including requests and response times, time requests wait in the queue, size of requests (bytes), and so forth.
kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=ClusterLink
Provides metrics on delayed operations on the cluster link.

Metrics on the Destination Cluster¶

Starting with Confluent Platform 7.0.0, the following broker metrics, specific to Cluster Linking, are available on the brokers in the destination cluster.

kafka.server:type=cluster-link-metrics,name=mirror-partition-count,link-name={linkName}
Number of actively mirrored partitions for which the broker is leader
kafka.server.link:type=ClusterLinkFetcherManager,name=FailedPartitionsCount,clientId=ClusterLink,link-name={linkName}
Number of failed mirror partitions for which the broker is leader. A partition may be in a failed state if the source partition’s epoch has gone backwards; for example, if the topic was recreated.
kafka.server:type=ReplicaManager,name=UnderMinIsrMirrorPartitionCount
Number of mirrored partitions that are under min ISR.
kafka.server:type=ReplicaManager,name=UnderReplicatedMirrorPartitions
Number of mirrored partitions that are under replicated.
kafka.server:type=cluster-link-metrics,name=(linked-leader-epoch-change-rate, linked-leader-epoch-change-total),link-name={linkName}

Rate per second and total number of times leader election was triggered on this broker due to source leader changes.

  • Frequent triggers for leader election might indicate issues on the source cluster. This can be a useful metric during on-premises to Confluent Cloud migrations to identify if there are issues on the source cluster.
kafka.server:type=cluster-link-metrics,name=(linked-topic-partition-addition-rate, linked-topic-partition-addition-total),link-name={linkName}

Rate per second and total number of times at which the partition count was updated due to source changes.

  • A high volume or rate of partition count updates might indicate issues on the source cluster.
kafka.server:type=cluster-link-metrics,name=(consumer-offset-committed-rate, consumer-offset-committed-total),link-name={linkName}

Rate per second and total number of consumer offsets committed for a cluster link.

  • Helps verify if offset migration is happening correctly, a critical metric for failover readiness scenarios.
kafka.server:type=cluster-link-metrics,name=(topic-config-update-rate, topic-config-update-total),link-name={linkName}

Rate per second and total number of topic configurations updates due to source topic configuration changes.

  • Can help troubleshooting and debugging issues where configurations are not propagated.
kafka.server:type=cluster-link-metrics,name=(acls-added-rate, acls-added-total),link-name={linkName}
Rate per second and total number of access control lists (ACLs) that are being added for each link.
kafka.server:type=cluster-link-metrics,name=(acls-deleted-rate, acls-deleted-total),link-name={linkName}
Rate per second and total number of access control lists (ACLs) that are being deleted for each link.
kafka.server:type=cluster-link-metrics,name=auto-mirror-created-total,link-name={linkName}
Total number of mirror topics that have been created with auto mirror topic creation.
kafka.server:type=cluster-link-metrics,name= auto-mirror-create-failed-total,link-name={linkName}
Total number of mirror topics that have failed to be created with auto mirror topic creation. This is indicative of a configuration issue with auto mirror topic creation
kafka.server:type=cluster-link-metrics,state=Mirror,link-name={link_name},name=mirror-topic-count

Total number of active (healthy) mirror topics.

Note

Note: Known issue: This metric will include topics that are in the SOURCE_UNAVAILABLE state. As a temporary workaround, use kafka.server:type=cluster-link-metrics,mode={mode},state=active,link-name={linkName} and kafka.server:type=cluster-link-metrics,mode={mode},state=unavailable,link-name={linkName} to alert on the source absence.

kafka.server:type=cluster-link-metrics,mode={mode},state=active,link-name={linkName}
Total number of active cluster links connected to the cluster. You can filter or group by direction (source or destination).
kafka.server:type=cluster-link-metrics,mode={mode},state=paused,link-name={linkName}
Total number of paused cluster links connected to the cluster. You can filter or group by direction (source or destination).
kafka.server:type=cluster-link-metrics,mode={mode},state=unavailable,link-name={linkName}
Total number of unavailable cluster links connected to the cluster. You can filter or group by direction (source or destination).
kafka.server:type=cluster-link-metrics,state=PausedMirror,link-name={link_name},name=mirror-topic-count
Total number of mirror topics for which mirroring from the source has been paused.
kafka.server:type=cluster-link-metrics,state=PendingStoppedMirror,link-name={link_name},name=mirror-topic-count
Total number of mirror topics whose mirroring has been temporarily stopped.
kafka.server:type=cluster-link-metrics,state=FailedMirror,link-name={link_name},name=mirror-topic-count
Total number of failed mirror topics; that is, mirroring cannot proceed.
kafka.server:type=cluster-link-metrics,state=active,link-name={link_name},name=link-count
Total number of links in an active state.
kafka.server:type=cluster-link-metrics,state=paused,link-name={link_name},name=link-count
Total number of links in a paused state.
kafka.server:type=cluster-link-metrics,state=unavailable,link-name={link_name},name=link-count
Total number of links in an unavailable state, where the source may not be available the link is having trouble fetching.
kafka.server:type=cluster-link-metrics,link-name={link_name},name=broker-failed-link-count
Total number of links in a failed state. This metric is reported on a per broker state, as some or all brokers could have a link in a failed state and unable to connect to the source.

Metrics on the Source Cluster¶

The source cluster (with Kafka or Confluent Server previous to 6.1.0) is unaware of cluster links, but can monitor cluster link related usage by allocating link-specific credentials. Quota metrics for each credential is available in both Kafka and Confluent Platform when user quotas are enabled.

kafka.server:type=Fetch,user={linkPrincipal},client-id={optionalLinkClientId}
  • byte-rate
  • throttle-time
kafka.server:type=Request,user={linkPrincipal},client-id={optionalLinkClientId}
  • request-time
  • throttle-time

If the source cluster is Confluent Server with a version of 6.1.0 or higher, then you can also monitor cluster links on the source cluster with the following metrics.

kafka.server:type=cluster-link-source-metrics,request={request},link-id={linkUUID}
  • request-byte-total
  • request-total
  • response-byte-total
  • response-time-ns-avg

Setting Quotas on Resources and Bandwidth Usage¶

You can set various types of quotas to place caps on resources and bandwidth used for Cluster Linking on the source and destination clusters.

Destination Cluster Quotas¶

You can limit total usage for cluster links on each broker in the destination cluster by setting the broker config confluent.cluster.link.io.max.bytes.per.second.

Source Cluster Quotas¶

The source cluster is unaware of cluster links, but can limit usage for cluster links by allocating link-specific credentials and assigning quotas for the link principal.

  • Fetch byte-rate (replication throughput) for each cluster link principal
  • Request rate quota (CPU/thread usage) for each cluster link principal. This also includes CPU usage for metadata and configuration sync as well as ACL and consumer offset migration.

Suggested Reading¶

  • See Monitoring Kafka with JMX for a full list of metrics for Confluent Platform.
  • See the Apache Kafka® documentation on metrics reporting and monitoring through JMX endpoints.

Was this doc page helpful?

Give us feedback

Do you still need help?

Confluent support portal Ask the community
Thank you. We'll be in touch!
Be the first to get updates and new content

By clicking "SIGN UP" you agree that your personal data will be processed in accordance with our Privacy Policy.

  • Confluent
  • About
  • Careers
  • Contact
  • Professional Services
  • Product
  • Confluent Cloud
  • Confluent Platform
  • Connectors
  • Flink
  • Stream Governance
  • Developer
  • Free Courses
  • Tutorials
  • Event Streaming Patterns
  • Documentation
  • Blog
  • Podcast
  • Community
  • Forum
  • Meetups
  • Kafka Summit
  • Catalysts
Terms & Conditions Privacy Policy Do Not Sell My Information Modern Slavery Policy Cookie Settings Feedback

Copyright © Confluent, Inc. 2014- Apache®️, Apache Kafka®️, Kafka®️, Apache Flink®️, Flink®️, Apache Iceberg®️, Iceberg®️ and associated open source project names are trademarks of the Apache Software Foundation

On this page: