Architecture Patterns for Multi-Region Clusters¶
Overview¶
Confluent Cloud and Confluent Platform are often used across multiple regions (data centers), for example for disaster recovery, migrations or geographic locality. Confluent Cloud and Confluent Platform contain several features and products that support multi-region architectures.
The following sections describe common multi-region architectures along with use cases and recommended high-level implementations.
Multi-region architectures require deep event streaming knowledge to understand and reason about. Please contact Confluent for guidance when considering these architectures.
Terms¶
| Term | Description |
|---|---|
| Recovery Point Objective (RPO) | In the event of failure, at which point in the data’s history does the failover need to resume from? In other words, how much data can be lost during a failure? In order to have zero RPO, synchronous replication is required. |
| Recovery Time Objective (RTO) | In the event of failure, how much time can elapse while a failover takes place? In other words, how long can a failover take? In order to have zero RTO, seamless client failover is required. |
| Region | A synonym for a data center. A fully operational region can run all of an application’s infrastructure. A light region can only run a very small amount of infrastructure, for example just one ZooKeeper host. |
| End user | The humans or computers that use the application that is using Confluent. |
| Application | The software that provides an interface to the end user and interacts with Confluent Cloud or Confluent Platform on the backend. |
| Disaster Recovery (DR) | Umbrella term that encompasses architecture, implementation, tooling, policies, and procedures that all allow an application to recover from a disaster, and in the context of this document, a full region failure. |
| High Availability (HA) | A highly available system can operate continuously even amidst failure. In the context of multi-region architectures, a highly available application built on Confluent can operate even during a full region failure. An HA application has a disaster recovery policy. |
| Event | A single message produced or consumed to/from Confluent Cloud or Confluent Platform. |
| Millisecond (ms) | 1/1,000th of a second |
In addition to the above terminology, review the multi-region concepts in Multi-Region Clusters.
Why Multi-Region?¶
An application may benefit from a multi-region architecture for one or more of the following reasons:
- Region failure disaster recovery - When a region experiences a region-wide failure, a multi-region architecture allows failover to a different region to decrease RTO and RPO.
- Global operations with minimized latency - End users in one locality, for example in North America, will connect to a North American region, whereas end users in another locality, for example in Europe, will connect to a European region. This ensures each end user will experience the lowest possible latency.
- Data sovereignty - End-user data within certain sovereignties must never leave the sovereignty or must be aggregated or anonymized when leaving the sovereignty.
- Data governance - Similar to data sovereignty, some applications require that certain data never leave a certain region or network but perhaps not because of a sovereignty’s laws.
- Network isolation - Some applications use Confluent Platform within or through a network DMZ to increase network security.
Why not Multi-Region?¶
Not all applications require a multi-region architecture. For example, many startups don’t have SLAs that require low RPO and/or low RTO during a full region failure.
Further, single-region architectures are simpler and easier to operate and maintain.
Multi-Cluster Replication (Mirroring)¶
In a multi-cluster replication architecture, some or all events in a region are replicated (mirrored) to another region. Sometimes these regions are called the source and destination, respectively.
This architecture is not attempting to decrease RPO or RTO. Instead, this architecture is required for a number of different use cases such as migrations. Technically, the 2-region architectures described above are multi-cluster replication architectures as well.
Refer to the use cases for this architecture to learn scenarios when this architecture is ideal.
When to use?¶
Use this architecture in one of the use cases below.
When not to use?¶
Do not use this architecture when a multi-region architecture is desired for high availability and disaster recovery.
Use Cases¶
Self-managed Confluent Platform to Confluent Cloud Migration¶
When migrating an application’s Confluent usage from a self-managed Confluent Platform deployment to Confluent Cloud, use a replication architecture to migrate events and components of the application in a piecemeal fashion to Confluent Cloud. This piecemeal method allows for the migration to occur in smaller, more manageable chunks, and possibly decreases or removes any downtime that might have otherwise been caused by a migration.
Lift and Shift Migration¶
A lift-and-shift migration is very similar to a Self-managed Confluent Platform to Confluent Cloud Migration, except that the original application might be using an alternative messaging system.
In this use case, rather than using replication components such as Cluster Linking, Replicator, or MirrorMaker, which all replicate events from one Confluent Platform cluster to another, use a source Connector or Bridge designed to read from the alternative messaging system and produce to the Confluent cluster.
Network Isolation¶
Some applications require that certain events only be available within certain regions or networks, whereas other events can or must be available in separate regions or networks.
In such applications, often a separate Confluent cluster will be used in each region or network, one cluster replicating to the other.
Data Governance¶
The Data Governance use case is very similar to the Network Isolation use case above, except that events may be restricted in certain regions for reasons other than network security. See also the Aggregation architecture below.
Implementation Recommendations¶
- Use Cluster Linking.
2-Region Active-Passive¶
A 2-region active-passive architecture involves two fully-operational regions, each running a separate Confluent cluster, where one cluster is the “active” cluster serving all produce and consume requests and the second “passive” cluster is a copy of the “active” but without applications running against it. When the “active” region fails, applications failover to the “passive” region. The key difference between this and the 2-region active-active architecture is that the active-passive architecture has applications only running in one region during normal operating conditions.
This architecture provides RPO>0 and RTO>0. RPO is greater than 0 because cross-region replication is asynchronous. RTO is greater than 0 because applications need to failover to a different Confluent cluster.
In this architecture, the entire application is run within the active region, producing and consuming from the local region. Every event produced to the local active region is replicated asynchronous to the passive backup region.
Then, when failover is triggered, applications are started in the passive region, producing and consuming from the passive region.
When to use?¶
Consider using this architecture when only two regions are available, when network latency is greater than 50ms, or when the application must only run in one region during normal operating conditions.
When not to use?¶
Do not use this architecture when RTO and/or RPO must be at or near 0.
Use Cases¶
On-premises with Limited Regions¶
Often this architecture is chosen by on-premises customers who have limited access to multiple regions. For example, some customers have one primary (the “active”) region and a smaller (the “passive”) region that is only used in disaster recovery scenarios.
Short-term Disaster Recovery¶
Sometimes startups or early Confluent adopters choose this architecture as a short-term disaster recovery solution, buying them time to implement a better architecture with lower RTO and/or RPO.
Implementation Recommendations¶
Use Cluster Linking.
2-Region Active-Active¶
A 2-region active-active architecture involves two fully-operational regions, each running a separate Confluent cluster, where each cluster is a copy of the other. When a region fails, applications failover to the other region.
This architecture provides RPO>0 and RTO~0. RPO is greater than 0 because cross-region replication is asynchronous. RTO can be very low because you have at least one region running at all times. Your architecture must detect a failure and diverts traffic upstream (such as through a load balancer or DNS) to the remaining active cluster. Applications must be deployed in both fully-operational regions in order to achieve an RTO close to 0.
Applications must be deployed in both fully operational regions in order to achieve an RTO=0. In most cases, each end-user connects to just one region, for example with geographic DNS routing. Then, in the event of a region failure, end-users are sent to the other, still active region.
When to use?¶
Consider using this architecture when only two fully operational regions are available, or when inter-region network latency is greater than 50ms.
Further, only consider this architecture when the application can be run in two regions simultaneously.
When not to use?¶
Do not use this architecture when an application can not run actively in two regions, perhaps because one region has less capacity than the other, or perhaps because two copies of the application cannot be run at once for legacy or functionality reasons.
Use Cases¶
Consumer Banking¶
Many consumer banks have low or zero RPO and RTO requirements and customers only within a single country or continent. However, for consumer banks running on-premises, sometimes only two fully-operational regions are available, making this architecture ideal.
Telecommunications¶
Telecommunications tend to have very similar characteristics to consumer banks: low or zero RPO and RTO, customers within a single country or continent, and access to only two fully-operational regions.
Implementation Recommendations¶
Use Cluster Linking.
Aggregation (Hub and Spoke)¶
An aggregation architecture has one or many regions, each running its own local Confluent cluster and application. All or a subset of the local events are replicated to a single, probably larger, aggregate Confluent cluster. Sometimes the aggregate cluster is referred to as the “hub,” and the other regions are referred to as “spokes.”
This architecture provides RPO>0 and RTO>=0. RPO is greater than 0 because cross-region replication is asynchronous. RTO is zero when end-users can be automatically redirected to another region. Otherwise, RTO is greater than 0.
Aggregation architectures are generally used in one of the following categories:
To support global operations where each spoke region is serving customers in the local geography, yet where a global view of all events is required as well.
To support data governance, where each team uses an independent Confluent cluster, replicating events to an aggregate cluster for applications that require cross-team events. Note that these clusters could run in the same region or separate regions.
When to use?¶
For global applications, only use this architecture when each application instance can operate independently of the others, when end-user latency must be low, when data centers are greater than 50ms of network latency apart, and when events must be aggregated, for example for analytics.
For data governance reasons, only use this architecture when truly separate clusters are required to satisfy the application’s data governance requirements.
When not to use?¶
Do not use this architecture when an application is not truly global. Also, do not use this architecture when Confluent’s native data governance functionality is satisfactory for the application’s data governance requirements.
Use Cases¶
Digital Advertising Click and Impression Tracking¶
The spoke regions run regional ad-serving and ad-tracking applications with a local Confluent cluster. Spoke clusters all replicate clicks and impressions data to an aggregate, hub region, running a larger Confluent cluster, where billing analytics are computed.
Online Gaming¶
Many online gaming use cases are similar to the Digital Advertising Click and Impression Tracking use case, except that the spoke regions are running game servers and the hub cluster is performing game analytics.
Brick-and-Mortar and Online Retail¶
Retailers with brick-and-mortar stores and e-commerce often operate a Confluent Cluster in each store (the “spokes”), supporting local applications. Events are replicated to and from a hub cluster, for example for analytics and inventory updates.
Hospitality¶
Similar to Retail above, hospitality providers often operate a Confluent cluster onsite to support local applications (the “spokes”), and a hub cluster for analytics.
Implementation Recommendations¶
Use Cluster Linking to replicate events from the spoke clusters to the hub cluster.
Stretched Cluster 3-Region¶
A stretched 3-region cluster architecture involves three regions that are all within 50ms of network latency (according to ping). Confluent Server and ZooKeeper are spread evenly across the three regions to make a single, stretched cluster.
Stretched clusters provide RPO=0 and RTO=0.
The most common environment where stretched clusters are deployed is in public cloud regions with three availability zones (AZ). In this case, each availability zone is considered a pseudo region, allowing the Confluent cluster to be available amidst a single AZ failure. However, if the entire region experiences a failure, the Confluent cluster may be unavailable.
Stretched cluster architectures are used outside of public clouds as well, and are sometimes used across public cloud regions. For example, in the United States, and either in public cloud or on-premises, regions in West, Central, and East may make up a single stretched cluster. In this case, the Confluent cluster will remain available amidst a full region failure.
Confluent Cloud standard, and dedicated clusters can be provisioned as stretched clusters, stretched across three AZs within a region.
When to use?¶
- Use a stretched cluster when three fully-operational regions can run Confluent, with a maximum of 50ms of network latency between all regions.
- Use a stretched cluster when the application only needs to survive a single region failure at a time. Stretched cluster architectures cannot survive two full-region failures because the ZooKeeper quorum will be incomplete.
- Stretched cluster architectures are most commonly used for applications located within a single country or smaller continent.
When not to use?¶
- Do not use a stretched cluster when regions are greater than 50ms of network latency apart, for example inter-continent.
- Do not use a stretched cluster when fewer than three fully-operational regions are available.
Use Cases¶
Consumer Banking¶
Many consumer banks have low or zero RPO and RTO requirements and customers only within a single country or continent. Often these consumer banks, for applications that are consumer facing especially, stretch a cluster across three regions spread across a country or continent. For example, in the United States, regions are often in the West, Central, and East.
Tip
To learn more about how to achieve near zero RPO and RTO, see the following blog post, which supplements the use cases detailed below: Automatic Observer Promotion Brings Fast and Safe Multi-Datacenter Failover with Confluent Platform 6.1
Telecommunications¶
Typically, telecommunications have very similar characteristics to consumer banks; low or zero RPO and RTO and customers within a single country or continent. These characteristics make a stretched cluster a great architecture.
Tip
To learn more about how to achieve near zero RPO and RTO, see the following blog post, which supplements the use cases detailed below: Automatic Observer Promotion Brings Fast and Safe Multi-Datacenter Failover with Confluent Platform 6.1
Implementation Recommendations¶
Deploy Confluent with rack awareness configured and Confluent Server and ZooKeeper hosts spread evenly across regions. Rack awareness ensures that replica placement is such that at least one topic-partition replica exists in each region (or rack).
In a five-host ZooKeeper deployment, two regions will have two ZooKeeper hosts, and the third region will have one ZooKeeper host.
Architecture Diagram¶
Stretched Cluster 2.5-Region (Confluent Platform Only)¶
A stretched 2.5-region architecture involves two fully-operational regions and one light (0.5) region running a single, stretched cluster. The fully operational regions run an equal number of Confluent Server and ZooKeeper hosts, whereas the light region runs a single ZooKeeper host. When any single region fails, the ZooKeeper quorum will remain available. When a fully operational region fails, applications failover to the other region.
This architecture provides RPO =0 or >0 depending on configuration, and RTO=>0. RPO is greater than 0 when inter-region replication is not guaranteed to be synchronous.
RTO is near zero when an application instance is running in each region, and you have automatic-observer promotion with replicas in each region, and a third datacenter with at least one Zookeeper, as described this blog post: Automatic Observer Promotion Brings Fast and Safe Multi-Datacenter Failover with Confluent Platform. You can also achieve RTO~0 if you have three or more datacenters all with replicas, even without observer promotion. Otherwise, RTO is greater than 0 due to needing to failover applications to another region.
You can achieve different RPO depending on replica configurations.
To achieve RPO=0, you need at least two regions with replicas, and the number of minimum in-sync replicas
must be greater than the number of replicas in any given region.
For example, if you configure two followers and one observer per location and min.ISR=3, there
is synchronous replication across the two locations, and RPO=0 for a datacenter failure.
Note that producers must be configured to acks=all support this. If producers are not adequately configured,
the replica placement as described is necessary but not sufficient to achieve the RPO=0.
Alternatively, with observers in the second datacenter only, there are no guarantees that all the data will be replicated to the surviving second datacenter if the first one fails.
When to use?¶
Consider using this architecture when only two fully operational regions are available and a light region is available. All regions must be within 50ms of network latency.
When not to use?¶
Do not use this architecture when regions are greater than 50ms apart, or when only two regions total are available.
Use Cases¶
Consumer Banking¶
Many consumer banks have low or zero RPO and RTO requirements and customers only within a single country or continent. However, for consumer banks running on-prem, sometimes three fully-operational regions are not available. As such, this architecture is a great option. Commonly the third, light region is a public cloud region. Most Infosec teams are comfortable with ZooKeeper running in a public cloud because ZooKeeper only contains metadata about the Confluent cluster.
Telecommunications¶
Typically, telecommunications have very similar characteristics to consumer banks: low or zero RPO and RTO, customers within a single country or continent, and access to only two fully-operational regions and one light region.
Implementation Recommendations¶
In all three regions, deploy ZooKeeper. In a five-host ZooKeeper cluster, deploy two in one fully-operational region, deploy two in the other fully-operational region, and deploy the fifth in the light region. Deploy an equal number of Confluent Server hosts in each fully-operational region.
Enable and configure the Multi-Region Clusters, including self-promoting observers. Create topics such that each topic has replicas in one region and observers in the other.
When one of the fully-operational regions fails, if applications are running in both regions, failover will happen automatically. However, if certain applications are running in the failed region, depending on the extent of the failure, those applications may need to be restarted in the second region.
Note that MRC is only available in Confluent Platform.
Architecture Diagram¶
Which architecture is best for my use case?¶
Use the below decision tree to understand what architecture is best for you.
As previously mentioned, multi-region architectures require deep event streaming knowledge to understand and reason about. You are welcome to contact Confluent for guidance when considering these architectures.
Comparison Chart of Multi-Region Options¶
| Product | Confluent Cloud Support | RPO | RTO | Description |
|---|---|---|---|---|
| Cluster Linking | Yes, self-managed connecting to Confluent Cloud | > 0 | > 0 | Cluster Linking perfectly mirrors topics and metadata from one cluster to another, and comes built-in to Confluent Server brokers |
| MirrorMaker | Yes, self-managed connecting to Confluent Cloud | > 0 | > 0 | KIP-382 |
| Multi-Region Clusters | No | = 0 or > 0, dependent on the configuration | >= 0 | Multi-Region Clusters (MRC) is a feature set that supports seamless failover and simpler multi-region architectures. |