Intelligent Replication Overview

Intelligent Replication is a performance optimization feature available exclusively in Confluent Private Cloud that delivers up to 10X or more performance improvements for Kafka workloads at scale. This feature leverages Confluent’s proven technology from Confluent Cloud, bringing advanced broker capabilities to self-managed environments.

What is Intelligent Replication?

Intelligent Replication includes two key components:

  • New Replication Implementation: Intelligent Replication introduces a new push-based replication mode for Confluent Private Cloud that complements Kafka’s existing pull-based replication. In traditional pull-based replication, follower brokers continuously fetch data from leader brokers. With push-based replication, leader brokers proactively send data to followers, reducing CPU overhead and improving efficiency.
  • Intelligent Algorithm: An algorithm that monitors and intelligently switches between PUSH and PULL based replication to achieve optimal performance while maintaining replica consistency. The system automatically transitions replicas from pull to push mode when they are caught up and in-sync, and falls back to pull mode when needed for recovery scenarios.

Key benefits

Up to 10X Performance at Scale
Initial benchmarks show massive improvements in p99 end-to-end latency for high-throughput workloads, even when scaling to 90,000+ partitions in a single cluster.
Automated Experience
Intelligent Replication monitors system performance and automatically adjusts replication to optimize performance and stability.
Predictable Latency Scaling
Latency remains consistent as throughput increases, unlike traditional Kafka where latency grows exponentially with throughput at high scale.
Reduced Infrastructure Costs
Lower CPU usage means you can handle more load with the same hardware.
Simplified Operations
One configuration parameter to enable the feature.

How it works

Intelligent Replication operates in a mixed-mode approach:

Initial State
All partition replicas start in pull mode using traditional Kafka replication.
Transition to Push Mode
When a follower replica is caught up to the leader and joins the In-Sync Replica (ISR) set, the leader automatically transitions it to push mode. The leader then proactively sends new data to the follower using AppendRecords requests.
Fallback to Pull Mode
If issues occur (such as network problems, follower lag, or errors), the system automatically falls back to pull mode to ensure data consistency and availability.
Session Management
The system uses replication session IDs to coordinate transitions between push and pull modes, ensuring proper ordering and preventing data loss.

Limitations

Observers and Intelligent Replication
Observers are not compatible with Intelligent Replication. If a cluster includes Observers, such as in Multi-Region clusters, enabling Intelligent Replication has no effect on Observer nodes. Observers continue to use their existing replication mechanism and ignore the Intelligent Replication configuration.
Confluent Private Cloud Only
This feature is only available in Confluent Private Cloud.

Compatibility

Intelligent Replication is backward compatible. No changes are required to Kafka clients, applications, or existing tools. The performance enhancements are managed entirely within the broker infrastructure.