# Resilience in Confluent Cloud This topic describes resilience features built into Apache Kafka® and how Confluent implements those features in Confluent Cloud. ## Kafka resilience Kafka is a distributed collection of servers, known as *brokers*, that operate as a [cluster](../_glossary.md#term-Kafka-cluster). Brokers can span data centers or cloud provider availability zones (AZs). Kafka clusters provide highly scalable and fault-tolerant systems that support mission-critical applications. *Partitions* distribute data across brokers in a cluster. You can specify how many *replicas* exist for each partition. One replica is the *leader* and the rest are *followers*. The leader handles read and write operations for the Kafka client, replicating write operations to followers. You can increase the operational reliability of the cluster by enforcing a minimum number of replicas that must be in sync to allow write operations and using client configurations such as [acks](/platform/current/installation/configuration/producer-configs.html#acks). If any broker in a cluster fails, the remaining brokers ensure continuous operations. In the event of a failure, a cluster rebalance event triggers leader elections to replace the lost leaders. You must configure Kafka clients to automatically handle rebalance events. These are the most fundamental aspects of Kafka resilience, which provides the native capability to operate in environments that require fault tolerance and scalability. ## Confluent Cloud resilience Confluent Cloud architecture provides two distinct decoupled pieces to separate points of failure: - A centralized global control plane - A data plane made up of satellite instances running in regions around the world Data planes function independently of the control plane and other data plane instances. Data plane instances are not affected by failures on the control plane or peer data plane instances. Confluent Cloud monitors and records event streams to data planes. This monitoring enables Confluent Cloud to determine when data planes have issues and to know what changes have occurred over time, so that Confluent Cloud can replay events if necessary. To enable communication between the control plane and the data planes, Confluent Cloud uses a special Kafka cluster known as the mothership Kafka, also called the mothership.  The mothership sends control messages to the data planes and forwards status updates from the data planes back to the control plane. Confluent uses a change data capture (CDC) approach, where Debezium connectors capture row-level changes committed to a Postgres database. Debezium connectors send those changes to the mothership, which refines them and sends them back to microservices running in the data plane. The microservice manages state in a relational store (with its ACID guarantees) and notifies other services of change. This approach resembles the Outbox Pattern. For Confluent this strategy is key for reliable, recoverable, and auditable communication from the control plane services to the data plane. ### Data plane Recovery point objective (RPO) is the amount of data you’re okay with losing because of a failure. To achieve zero RPO, you need synchronous replication, which Confluent Cloud provides. Recovery time objective (RTO) is the amount of time necessary for recovery from a failure. To have zero RTO, you must properly configure clients to fail over. #### RPO In a multi-zone cluster, Confluent Cloud provides a replication factor of three and enforces a minimum number of two in-sync replicas. This means Confluent Cloud replicates data across three cloud provider AZs and requires two replicas to be in sync to allow write operations. This configuration provides the synchronous replication required for zero RPO. Confluent Cloud continues to operate normally as long as two AZs are operational. In other words, Confluent Cloud provides fault tolerance for the loss of a single AZ.  The number of AZs lost determines how the cluster behaves: - Loss of one AZ: The cluster continues to serve reads and writes with no change to operations. - Loss of two AZs: The cluster loses write availability, because it can no longer meet the minimum of two in-sync replicas. Reads of existing data continue to work. - Loss of three AZs: The cluster loses both read and write availability. #### RTO Loss of an AZ triggers a cluster rebalance event. This process is immediate and automatic and ensures connected applications can access data. For a single AZ failure, RTO is less than one second for Kafka clients configured according to best practices. For more information, see [Producer Configurations](/platform/current/installation/configuration/producer-configs.html#ak-producer-configurations-for-cp) and [Kafka 101 Developer course section on Producers](https://developer.confluent.io/learn-kafka/apache-kafka/producers/). ## Related content - [Confluent Cloud Security Controls whitepaper (PDF)](https://assets.confluent.io/m/67f93edfe1a9f524/original/20200804-WP-Confluent_Cloud_Security_Controls.pdf) - [Optimize and Tune Confluent Cloud Clients](../client-apps/optimizing/overview.md#ccloud-optimizing)