<a id="intelligent-replication-overview"></a>

# Intelligent Replication Overview

Intelligent Replication is a performance optimization feature available
exclusively in Confluent Private Cloud that delivers up to 10X or more performance improvements
for Kafka workloads at scale. This feature leverages Confluent’s proven
technology from Confluent Cloud, bringing advanced broker capabilities to
self-managed environments.

## What is Intelligent Replication?

Intelligent Replication includes two key components:

- **New Replication Implementation**: Intelligent Replication introduces a new
  push-based replication mode for Confluent Private Cloud that complements Kafka’s existing
  pull-based replication. In traditional pull-based replication, follower
  brokers continuously fetch data from leader brokers. With push-based
  replication, leader brokers proactively send data to followers, reducing
  CPU overhead and improving efficiency.
- **Intelligent Algorithm**: An algorithm that monitors and intelligently
  switches between PUSH and PULL based replication to achieve optimal
  performance while maintaining replica consistency. The system automatically
  transitions replicas from pull to push mode when they are caught up and
  in-sync, and falls back to pull mode when needed for recovery scenarios.

## Key benefits

**Up to 10X Performance at Scale**
: Initial benchmarks show massive improvements in p99 end-to-end latency for
  high-throughput workloads, even when scaling to 90,000+ partitions in a
  single cluster.

**Automated Experience**
: Intelligent Replication monitors system performance and automatically
  adjusts replication to optimize performance and stability.

**Predictable Latency Scaling**
: Latency remains consistent as throughput increases, unlike traditional
  Kafka where latency grows exponentially with throughput at high scale.

**Reduced Infrastructure Costs**
: Lower CPU usage means you can handle more load with the same hardware.

**Simplified Operations**
: One configuration parameter to enable the feature.

## How it works

Intelligent Replication operates in a mixed-mode approach:

**Initial State**
: All partition replicas start in pull mode using traditional Kafka
  replication.

**Transition to Push Mode**
: When a follower replica is caught up to the leader and joins the In-Sync
  Replica (ISR) set, the leader automatically transitions it to push mode.
  The leader then proactively sends new data to the follower using
  `AppendRecords` requests.

**Fallback to Pull Mode**
: If issues occur (such as network problems, follower lag, or errors), the
  system automatically falls back to pull mode to ensure data consistency
  and availability.

**Session Management**
: The system uses replication session IDs to coordinate transitions between
  push and pull modes, ensuring proper ordering and preventing data loss.

## Limitations

**Observers and Intelligent Replication**
: Observers are not compatible with Intelligent Replication. If a cluster
  includes Observers, such as in Multi-Region clusters, enabling
  Intelligent Replication has no effect on Observer nodes. Observers continue
  to use their existing replication mechanism and ignore the Intelligent
  Replication configuration.

**Confluent Private Cloud Only**
: This feature is only available in Confluent Private Cloud. To use this feature, you must have a Confluent Enterprise license for Confluent Private Cloud subscription. For more information, see the [Manage Confluent Platform Licenses](../../installation/license.md#cp-license-overview) documentation.

## Compatibility

Intelligent Replication is backward compatible. No changes are required to
Kafka clients, applications, or existing tools. The performance enhancements
are managed entirely within the broker infrastructure.