# Tutorial: Multi-Region Clusters on Confluent Platform This tutorial describes the Multi-Region Clusters capability that is built directly into Confluent Server. Multi-Region Clusters allow customers to run a single Apache Kafka® cluster across multiple datacenters. Often referred to as a stretch cluster, Multi-Region Clusters replicate data between datacenters across regional availability zones. You can choose how to replicate data, synchronously or asynchronously, on a per Kafka topic basis. It provides good durability guarantees and makes disaster recovery (DR) much easier. Benefits: - Supports multi-site deployments of synchronous and asynchronous replication between datacenters - Consumers can leverage data locality for reading Kafka data, which means better performance and lower cost - Ordering of Kafka messages is preserved across datacenters - Consumer offsets are preserved - In event of a disaster in a datacenter, new leaders are automatically elected in the other datacenter for the topics configured for synchronous replication, and applications proceed without interruption, achieving very low RTOs and RPO=0 for those topics. ## Concepts `Replicas` are brokers assigned to a topic-partition, and they can be a *Leader*, *Follower*, or *Observer*. A *Leader* is the broker/replica accepting producer messages. A *Follower* is a broker/replica that can join an ISR list and participate in the calculation of the high watermark (used by the leader when acknowledging messages back to the producer). An `ISR` list (in-sync replicas) includes brokers that have a given topic-partition. The data is copied from the leader to every member of the ISR before the producer gets an acknowledgment. The followers in an ISR can become the leader if the current leader fails. An `Observer` is a broker/replica that also has a copy of data for a given topic-partition, and consumers are allowed to read from them even though the *Observer* isn’t the leader–this is known as “Follower Fetching”. However, the data is copied asynchronously from the leader such that a producer doesn’t wait on observers to get back an acknowledgment. ![Follower fetching](images/Follower_Fetching.png) In “non-degraded” steady state, observers don’t participate in the ISR list and won’t become the leader. If a broker in the ISR fails, observers could be promoted to the ISR list in one of two ways: manual changes to leader assignment, or automatically with `Automatic Observer Promotion`. `Automatic Observer Promotion` is the process by which an observer is promoted into the ISR in certain “degraded” situations. The qualifications for whether an observer can be automatically promoted into the ISR is controlled by the `observerPromotionPolicy` field in a topic’s replica placement policy: - `under-min-isr`: if the number of replicas in the ISR drops below the topic’s `min.insync.replicas` configuration. - `under-replicated`: if the number of replicas in the ISR drops below the configured count of replicas in the topic’s replica placement policy. - `leader-is-observer`: if the current partition leader is an observer. ## Configuration The scenario for this tutorial is as follows: - Three regions: `west`, `central`, and `east` - Broker naming convention: `broker-[region]-[broker_id]` ![Multi-region Architecture](images/multi-region-base.png) Note that this tutorial uses a separate Kafka cluster backing Confluent Control Center, which monitors the multi-region cluster. Here are some relevant configuration parameters at different component levels: ### Broker You can find full broker configurations in the [docker-compose.yml file](https://github.com/confluentinc/examples/tree/latest/multiregion/docker-compose.yml). The most important configuration parameters include: - `broker.rack`: identifies the location of the broker. For the example, it represents a region, either `east` or `west` - `replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector`: allows clients to read from followers (in contrast, clients are typically only allowed to read from leaders) - `confluent.log.placement.constraints`: sets the default replica placement constraint configuration for newly created topics. - `confluent.metrics.reporter.bootstrap.servers` directs metrics to the dedicated metrics cluster. ### Client - `client.rack`: identifies the location of the client. For the example, it represents a region, either `east` or `west` - `replication.factor`: at the topic level, the replication factor is mutually exclusive to replica placement constraints, so for Kafka Streams applications, set `replication.factor=-1` to let replica placement constraints take precedence - `min.insync.replicas`: durability guarantees are driven by replica placement and `min.insync.replicas`. The number of followers in each region should be sufficient to meet `min.insync.replicas`, for example, if `min.insync.replicas=3`, then `west` should have 3 replicas and `east` should have 3 replicas. ### Topic - `--replica-placement `: at topic creation, this argument defines the replica placement policy for a given topic ## Prerequisites - Install [Docker Desktop](https://docs.docker.com/desktop/) (version `4.0.0` or later) or [Docker Engine](https://docs.docker.com/engine/install/) (version `19.03.0` or later) if you don’t already have it. - Install the [Docker Compose plugin](https://docs.docker.com/compose/install/) if you don’t already have it. This isn’t necessary if you have Docker Desktop, since it includes Docker Compose. - Start Docker if it’s not already running, either by starting Docker Desktop or, if you manage Docker Engine with `systemd`, via [systemctl](https://docs.docker.com/config/daemon/systemd/). - Verify that Docker is set up properly by ensuring no errors are output when you run `docker info` and `docker compose version` on the command line. ## Download and run the tutorial 1. Clone the [confluentinc/examples](https://github.com/confluentinc/examples) GitHub repository, and check out the `8.3.1-post` branch. ```bash git clone https://github.com/confluentinc/examples ``` ```bash cd examples ``` ```bash git checkout 8.3.1-post ``` 2. Go to the directory with the Multi-Region Clusters by running the following command: ```text cd multiregion ``` 3. If you want to manually step through this tutorial, which is advised for new users who want to gain familiarity with Multi-Region Clusters, [skip ahead to the next section](#multiregion-manual-demo). Alternatively, you can run the full tutorial end-to-end with the following script, which automates all the steps in the tutorial: ```bash ./scripts/start.sh ``` The automated demo takes several minutes to run to completion and return your prompt. You can follow the progress by comparing the output with the workflow and code examples in the manual steps. Use Ctl-C if you want to stop the demo early. Be sure to stop all services and clean up the Docker environment after running this script, as described in [Stop the Tutorial and Teardown](#multiregion-demo-teardown). ## Startup 1. This Multi-Region Clusters example uses Traffic Control (`tc`) to inject latency between the regions and packet loss to simulate the WAN link. Confluent’s ubi-based Docker images do not have `tc` installed, so build a custom Docker image with `tc`. ```bash ./scripts/build_docker_image.sh ``` 2. Start all the Docker containers ```bash docker compose up -d ``` 3. You should see the following Docker containers with `docker compose ps`: ```text NAME COMMAND SERVICE CREATED STATUS PORTS broker-ccc "/etc/confluent/docker/run" broker-ccc 2 minutes ago Up 2 minutes 0.0.0.0:8099->8099/tcp, 0.0.0.0:9099->9099/tcp, 9092/tcp broker-east-3 "/etc/confluent/docker/run" broker-east-3 2 minutes ago Up 2 minutes 0.0.0.0:8093->8093/tcp, 0.0.0.0:9093->9093/tcp, 9092/tcp broker-east-4 "/etc/confluent/docker/run" broker-east-4 2 minutes ago Up 2 minutes 0.0.0.0:8094->8094/tcp, 0.0.0.0:9094->9094/tcp, 9092/tcp broker-west-1 "/etc/confluent/docker/run" broker-west-1 2 minutes ago Up 2 minutes 0.0.0.0:8091->8091/tcp, 0.0.0.0:9091->9091/tcp, 9092/tcp broker-west-2 "/etc/confluent/docker/run" broker-west-2 2 minutes ago Up 2 minutes 0.0.0.0:8092->8092/tcp, 0.0.0.0:9092->9092/tcp control-center "/etc/confluent/docker/run" control-center 2 minutes ago Up 2 minutes 0.0.0.0:9021->9021/tcp controller-ccc "/etc/confluent/docker/run" controller-ccc 2 minutes ago Up 2 minutes 0.0.0.0:8098->8098/tcp, 0.0.0.0:9098->9098/tcp, 9092/tcp controller-central "/etc/confluent/docker/run" controller-central 2 minutes ago Up 2 minutes 0.0.0.0:8096->8096/tcp, 0.0.0.0:9096->9096/tcp, 9092/tcp controller-east "/etc/confluent/docker/run" controller-east 2 minutes ago Up 2 minutes 0.0.0.0:8097->8097/tcp, 0.0.0.0:9097->9097/tcp, 9092/tcp controller-west "/etc/confluent/docker/run" controller-west 2 minutes ago Up 2 minutes 0.0.0.0:8095->8095/tcp, 0.0.0.0:9095->9095/tcp, 9092/tcp ``` ## Inject latency and packet loss Here is a diagram of the simulated latency between the regions and the WAN link. ![Multi-region latencies](images/multi-region-latencies.png) 1. View the IP addresses used by Docker for the example: ```text docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq) ``` 2. Run the script [latency_docker.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/latency_docker.sh) that configures `tc` on the Docker containers: ```bash ./scripts/latency_docker.sh ``` ## Replica Placement This tutorial demonstrates the principles of Multi-Region Clusters through various topics. ![Multi-region topic replicas](images/multi-region-topic-replicas.png) Each topic has a replica placement policy that specifies a set of matching constraints (for example, `count` and `rack` for `replicas` and `observers`). The replica placement policy file is defined with the argument `--replica-placement ` mentioned earlier (these files are in the [config](https://github.com/confluentinc/examples/tree/latest/multiregion/config/) directory). Each placement also has an associated minimum `count` that guarantees a certain spread of replicas throughout the cluster. In this tutorial, you will create the following topics. You could create all the topics by running the script [create-topics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/create-topics.sh), but we will step through each topic creation to demonstrate the required arguments. | Topic name | Leader | Followers (sync replicas) | Observers (async replicas) | ISR list | Use default placement constraints | Observer Promotion policy | |------------------------------------------|----------|-----------------------------|------------------------------|------------|-------------------------------------|-----------------------------| | single-region | 1x west | 1x west | n/a | {1,2} | no | none | | multi-region-sync | 1x west | 1x west, 2x east | n/a | {1,2,3,4} | no | none | | multi-region-async | 1x west | 1x west | 2x east | {1,2} | no | none | | multi-region-async-op-under-min-isr | 1x west | 1x west | 2x east | {1,2} | no | under-min-isr | | multi-region-async-op-under-replicated | 1x west | 1x west | 2x east | {1,2} | no | under-replicated | | multi-region-async-op-leader-is-observer | 1x west | 1x west | 2x east | {1,2} | no | leader-is-observer | | multi-region-default | 1x west | 1x west | 2x east | {1,2} | yes | none | 1. Create the Kafka topic `single-region`. ```none docker compose exec broker-west-1 kafka-topics --create \ --bootstrap-server broker-west-1:19091 \ --topic single-region \ --partitions 1 \ --replica-placement /etc/kafka/demo/placement-single-region.json \ --config min.insync.replicas=1 ``` Here is the topic’s replica placement policy [placement-single-region.json](https://github.com/confluentinc/examples/tree/latest/multiregion/config/placement-single-region.json): ```none { "version": 1, "replicas": [ { "count": 2, "constraints": { "rack": "west" } } ] } ``` 2. Create the Kafka topic `multi-region-sync`. ```none docker compose exec broker-west-1 kafka-topics --create \ --bootstrap-server broker-west-1:19091 \ --topic multi-region-sync \ --partitions 1 \ --replica-placement /etc/kafka/demo/placement-multi-region-sync.json \ --config min.insync.replicas=1 ``` Here is the topic’s replica placement policy [placement-multi-region-sync.json](https://github.com/confluentinc/examples/tree/latest/multiregion/config/placement-multi-region-sync.json): ```none { "version": 1, "replicas": [ { "count": 2, "constraints": { "rack": "west" } }, { "count": 2, "constraints": { "rack": "east" } } ] } ``` 3. Create the Kafka topic `multi-region-async`. ```none docker compose exec broker-west-1 kafka-topics --create \ --bootstrap-server broker-west-1:19091 \ --topic multi-region-async \ --partitions 1 \ --replica-placement /etc/kafka/demo/placement-multi-region-async.json \ --config min.insync.replicas=1 ``` Here is the topic’s replica placement policy [placement-multi-region-async.json](https://github.com/confluentinc/examples/tree/latest/multiregion/config/placement-multi-region-async.json): ```none { "version": 1, "replicas": [ { "count": 2, "constraints": { "rack": "west" } } ], "observers": [ { "count": 2, "constraints": { "rack": "east" } } ] } ``` 4. Create the Kafka topic `multi-region-async-op-under-min-isr`. ```none docker compose exec broker-west-1 kafka-topics \ --create \ --bootstrap-server broker-west-1:19091 \ --topic multi-region-async-op-under-min-isr \ --partitions 1 \ --replica-placement /etc/kafka/demo/placement-multi-region-async-op-under-min-isr.json \ --config min.insync.replicas=2 ``` Here is the topic’s replica placement policy [placement-multi-region-async-op-under-min-isr.json](https://github.com/confluentinc/examples/tree/latest/multiregion/config/placement-multi-region-async-op-under-min-isr.json): ```none { "version": 2, "replicas": [ { "count": 2, "constraints": { "rack": "west" } } ], "observers": [ { "count": 2, "constraints": { "rack": "east" } } ], "observerPromotionPolicy":"under-min-isr" } ``` 5. Create the Kafka topic `multi-region-async-op-under-replicated`. ```none docker compose exec broker-west-1 kafka-topics \ --create \ --bootstrap-server broker-west-1:19091 \ --topic multi-region-async-op-under-replicated \ --partitions 1 \ --replica-placement /etc/kafka/demo/placement-multi-region-async-op-under-replicated.json \ --config min.insync.replicas=1 ``` Here is the topic’s replica placement policy [placement-multi-region-async-op-under-replicated.json](https://github.com/confluentinc/examples/tree/latest/multiregion/config/placement-multi-region-async-op-under-replicated.json): ```none { "version": 2, "replicas": [ { "count": 2, "constraints": { "rack": "west" } } ], "observers": [ { "count": 2, "constraints": { "rack": "east" } } ], "observerPromotionPolicy":"under-replicated" } ``` 6. Create the Kafka topic `multi-region-async-op-leader-is-observer`. ```none docker compose exec broker-west-1 kafka-topics \ --create \ --bootstrap-server broker-west-1:19091 \ --topic multi-region-async-op-leader-is-observer \ --partitions 1 \ --replica-placement /etc/kafka/demo/placement-multi-region-async-op-leader-is-observer.json \ --config min.insync.replicas=1 ``` Here is the topic’s replica placement policy [placement-multi-region-async-op-leader-is-observer.json](https://github.com/confluentinc/examples/tree/latest/multiregion/config/placement-multi-region-async-op-leader-is-observer.json): ```none { "version": 2, "replicas": [ { "count": 2, "constraints": { "rack": "west" } } ], "observers": [ { "count": 2, "constraints": { "rack": "east" } } ], "observerPromotionPolicy":"leader-is-observer" } ``` 7. Create the Kafka topic `multi-region-default`. Note that the `--replica-placement` argument is not used in order to demonstrate the default placement constraints. ```none docker compose exec broker-west-1 kafka-topics \ --create \ --bootstrap-server broker-west-1:19091 \ --topic multi-region-default \ --config min.insync.replicas=1 ``` 8. View the topic replica placement by running the script [describe-topics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/describe-topics.sh): ```bash ./scripts/describe-topics.sh ``` You should see output similar to the following: ```text ==> Describe topic: single-region Topic: single-region PartitionCount: 1 ReplicationFactor: 2 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[]} Topic: single-region Partition: 0 Leader: 2 Replicas: 2,1 Isr: 2,1 Offline: ==> Describe topic: multi-region-sync Topic: multi-region-sync PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}},{"count":2,"constraints":{"rack":"east"}}],"observers":[]} Topic: multi-region-sync Partition: 0 Leader: 1 Replicas: 1,2,3,4 Isr: 1,2,3,4 Offline: ==> Describe topic: multi-region-async Topic: multi-region-async PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,1 Offline: Observers: 3,4 ==> Describe topic: multi-region-async-op-under-min-isr Topic: multi-region-async-op-under-min-isr PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=2,confluent.placement.constraints={"observerPromotionPolicy":"under-min-isr","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-under-min-isr Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,1 Offline: Observers: 3,4 ==> Describe topic: multi-region-async-op-under-replicated Topic: multi-region-async-op-under-replicated PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"observerPromotionPolicy":"under-replicated","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-under-replicated Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,1 Offline: Observers: 3,4 ==> Describe topic: multi-region-async-op-leader-is-observer Topic: multi-region-async-op-leader-is-observer PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"observerPromotionPolicy":"leader-is-observer","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-leader-is-observer Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,1 Offline: Observers: 3,4 ==> Describe topic: multi-region-default Topic: multi-region-default PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-default Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,1 Offline: Observers: 3,4 ``` 9. View the topic replica placement in Confluent Control Center: Navigate to the Confluent Control Center UI at [http://localhost:9021](http://localhost:9021). ![Control Center cluster navigation](images/c3-cluster-navigation.png) Notice two clusters: “mrc” which is the multiregion cluster, and “metrics” which is a dedicated metrics cluster that runs Confluent Control Center. By backing Confluent Control Center to its own Kafka cluster, it has no dependency on the availability of the production cluster it is monitoring. The remainder of this tutorial works on topics within the “mrc” cluster. Click on the “mrc” cluster, then make your way to the “Topics” section. ![Control Center topics overview](images/c3-topics-overview.png) Click on each topic to see details about the replica and observer placement. Confluent Control Center matches the CLI output above. Below is an example of the `multi-region-async` topic. ![multi-region-async replicas](images/multi-region-async-placement.png) 10. Observe the following: - The `multi-region-async`, `multi-region-async-op-under-min-isr`, `multi-region-async-op-under-replicated`, `multi-region-async-op-leader-is-observer` and `multi-region-default` topics have replicas across `west` and `east` regions, but only 1 and 2 are in the ISR, and 3 and 4 are observers. This can be observed via the CLI output or Confluent Control Center. ## Client Performance ### Producer 1. Run the producer perf test script [run-producer.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/run-producer.sh): ```bash ./scripts/run-producer.sh ``` 2. Verify that you see performance results similar to the following: ```text ==> Produce: Single-region Replication (topic: single-region) 5000 records sent, 240.453977 records/sec (1.15 MB/sec), 10766.48 ms avg latency, 17045.00 ms max latency, 11668 ms 50th, 16596 ms 95th, 16941 ms 99th, 17036 ms 99.9th. ==> Produce: Multi-region Sync Replication (topic: multi-region-sync) 100 records sent, 2.145923 records/sec (0.01 MB/sec), 34018.18 ms avg latency, 45705.00 ms max latency, 34772 ms 50th, 44815 ms 95th, 45705 ms 99th, 45705 ms 99.9th. ==> Produce: Multi-region Async Replication to Observers (topic: multi-region-async) 5000 records sent, 228.258388 records/sec (1.09 MB/sec), 11296.69 ms avg latency, 18325.00 ms max latency, 11866 ms 50th, 17937 ms 95th, 18238 ms 99th, 18316 ms 99.9th. ``` 3. Observe the following: - In the first and third cases, the `single-region` and `multi-region-async` topics have nearly the same throughput performance (for examples, `1.15 MB/sec` and `1.09 MB/sec`, respectively, in the previous example), because only the replicas in the `west` region need to acknowledge. - In the second case for the `multi-region-sync` topic, due to the poor network bandwidth between the `east` and `west` regions and to an ISR made up of brokers in both regions, it took a big throughput hit (for example, `0.01 MB/sec` in the previous example). This is because the producer is waiting for an `ack` from all members of the ISR before continuing, including those in `west` and `east`. - The observers in the third case for topic `multi-region-async` didn’t affect the overall producer throughput because the `west` region is sending an `ack` back to the producer after it has been replicated twice in the `west` region, and it is not waiting for the async copy to the `east` region. - This example doesn’t produce to `multi-region-default` because the behavior is the same as `multi-region-async` since the configuration is the same. ### Consumer 1. Run the consumer perf test script [run-consumer.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/run-consumer.sh), where the consumer is in `east`: ```bash ./scripts/run-consumer.sh ``` 2. Verify that you see performance results similar to the following: ```text ==> Consume from east: Multi-region Async Replication reading from Leader in west (topic: multi-region-async) start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec 2019-09-25 17:10:27:266, 2019-09-25 17:10:53:683, 23.8419, 0.9025, 5000, 189.2721, 1569431435702, -1569431409285, -0.0000, -0.0000 ==> Consume from east: Multi-region Async Replication reading from Observer in east (topic: multi-region-async) start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec 2019-09-25 17:10:56:844, 2019-09-25 17:11:02:902, 23.8419, 3.9356, 5000, 825.3549, 1569431461383, -1569431455325, -0.0000, -0.0000 ``` 3. Observe the following: - In the first scenario, the consumer running in `east` reads from the leader in `west` and is impacted by the low bandwidth between `east` and `west`. The throughput is lower in this case (for example, `0.9025` MB per sec in the previous example). - In the second scenario, the consumer running in `east` reads from the follower that is also in `east`–the throughput of the consumer is higher in this case (for example, `3.9356` MBps in the previous example). - This example doesn’t consume from `multi-region-default` as the behavior should be the same as `multi-region-async` since the configuration is the same. ## Monitoring In Confluent Server there are a few JMX metrics you should monitor for determining the health and state of a topic partition. The tutorial describes the following JMX metrics. For a description of other relevant JMX metrics, see [Multi-Region Clusters metrics](multi-region.md#mrr-metrics). - `ReplicasCount` - In JMX the full object name is `kafka.cluster:type=Partition,name=ReplicasCount,topic=,partition=`. It reports the number of replicas (sync replicas and observers) assigned to the topic partition. - `InSyncReplicasCount` - In JMX the full object name is `kafka.cluster:type=Partition,name=InSyncReplicasCount,topic=,partition=`. It reports the number of replicas in the ISR. - `CaughtUpReplicasCount` - In JMX the full object name is `kafka.cluster:type=Partition,name=CaughtUpReplicasCount,topic=,partition=`. It reports the number of replicas that are consider caught up to the topic partition leader. Note that this may be greater than the size of the ISR as observers may be caught up but are not part of ISR. - `ObserversInIsrCount` - In JMX the full object name is `kafka.cluster:type=Partition,name=ObserversInIsrCount,topic=,partition=`. It reports the number of observers that are currently promoted to the ISR. There is a script you can run to collect the JMX metrics from the command line, but the general form is: ```bash docker compose exec broker-west-1 kafka-run-class kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:8091/jmxrmi --object-name kafka.cluster:type=Partition,name=,topic=,partition=0 --one-time true ``` 1. Run the script [jmx_metrics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/jmx_metrics.sh) to get the JMX metrics for `ReplicasCount`, `InSyncReplicasCount`, `CaughtUpReplicasCount`, and `ObserversInIsrCount` from each of the brokers: ```bash ./scripts/jmx_metrics.sh ``` 2. Verify you see output similar to the following: ```text ==> JMX metric: ReplicasCount single-region: 2 multi-region-sync: 4 multi-region-async: 4 multi-region-async-op-under-min-isr: 4 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 4 multi-region-default: 4 ==> JMX metric: InSyncReplicasCount single-region: 2 multi-region-sync: 4 multi-region-async: 2 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 2 multi-region-default: 2 ==> JMX metric: CaughtUpReplicasCount single-region: 2 multi-region-sync: 4 multi-region-async: 4 multi-region-async-op-under-min-isr: 4 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 4 multi-region-default: 4 ==> JMX metric: ObserversInIsrCount single-region: 0 multi-region-sync: 0 multi-region-async: 0 multi-region-async-op-under-min-isr: 0 multi-region-async-op-under-replicated: 0 multi-region-async-op-leader-is-observer: 0 multi-region-default: 0 ``` ## Degraded Region In this section, you will simulate a single broker failure in the `west` region. 1. Run the following command to stop one of the broker Docker containers in the `west` region: ```bash docker compose stop controller-west-1 ``` 2. Verify the new topic replica placement by running the script [describe-topics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/describe-topics.sh): ```bash ./scripts/describe-topics.sh ``` You should see output similar to the following: ```text ==> Describe topic: single-region Topic: single-region PartitionCount: 1 ReplicationFactor: 2 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[]} Topic: single-region Partition: 0 Leader: 2 Replicas: 1,2 Isr: 2 Offline: 1 ==> Describe topic: multi-region-sync Topic: multi-region-sync PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}},{"count":2,"constraints":{"rack":"east"}}],"observers":[]} Topic: multi-region-sync Partition: 0 Leader: 2 Replicas: 1,2,3,4 Isr: 2,3,4 Offline: 1 ==> Describe topic: multi-region-async Topic: multi-region-async PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async Partition: 0 Leader: 2 Replicas: 1,2,4,3 Isr: 2 Offline: 1 Observers: 4,3 ==> Describe topic: multi-region-async-op-under-min-isr Topic: multi-region-async-op-under-min-isr PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=2,confluent.placement.constraints={"observerPromotionPolicy":"under-min-isr","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-under-min-isr Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,4 Offline: 1 Observers: 3,4 ==> Describe topic: multi-region-async-op-under-replicated Topic: multi-region-async-op-under-replicated PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"observerPromotionPolicy":"under-replicated","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-under-replicated Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,4 Offline: 1 Observers: 3,4 ==> Describe topic: multi-region-async-op-leader-is-observer Topic: multi-region-async-op-leader-is-observer PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"observerPromotionPolicy":"leader-is-observer","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-leader-is-observer Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2 Offline: 1 Observers: 3,4 ==> Describe topic: multi-region-default Topic: multi-region-default PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-default Partition: 0 Leader: 2 Replicas: 1,2,3,4 Isr: 2 Offline: 1 Observers: 3,4 ``` 3. Verify similar replica placement in Confluent Control Center. Note that it may take up to 5 minutes it to properly report the new topic stats. ![Control Center degraded region](images/c3-degraded-region.png) 4. Observe the following: - In all topics except `multi-region-async-op-under-min-isr`, `multi-region-sync` and `multi-region-async-op-under-replicated` there is only 1 replica in the ISR. This is because replica placement dictated all replicas were in the `west` region which has only 1 remaining live broker. - In the second scenario, the `multi-region-sync` topic maintained an ISR of 3 brokers. This is because its placement policy always allows for brokers from east to join the ISR. - The `multi-region-async-op-under-min-isr` and `multi-region-async-op-under-replicated` topics have placement policies that allow observers to be automatically promoted into the ISR. In the case of `multi-region-async-op-under-min-isr` the number of non-observer replicas (1) is less than the `min.insync.replicas` value (2). Observers are promoted to the ISR to meet the `min.insync.replicas` requirement. In the case of `multi-region-async-op-under-replicated` the number of online replicas (1) is less than the intended number of non-observer replicas from the replica placement (2). An observer is promoted to fulfill this requirement. 5. Run the script [jmx_metrics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/jmx_metrics.sh) to get the JMX metrics for `ReplicasCount`, `InSyncReplicasCount`, `CaughtUpReplicasCount`, and `ObserversInIsrCount` from each of the brokers: ```bash ./scripts/jmx_metrics.sh ``` 6. Verify you see output similar to the following: ```text ==> JMX metric: ReplicasCount single-region: 2 multi-region-sync: 4 multi-region-async: 4 multi-region-async-op-under-min-isr: 4 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 4 multi-region-default: 4 ==> JMX metric: InSyncReplicasCount single-region: 1 multi-region-sync: 3 multi-region-async: 1 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 1 multi-region-default: 1 ==> JMX metric: CaughtUpReplicasCount single-region: 1 multi-region-sync: 4 multi-region-async: 3 multi-region-async-op-under-min-isr: 3 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 4 multi-region-default: 3 ==> JMX metric: ObserversInIsrCount single-region: 0 multi-region-sync: 0 multi-region-async: 0 multi-region-async-op-under-min-isr: 1 multi-region-async-op-under-replicated: 1 multi-region-async-op-leader-is-observer: 0 multi-region-default: 0 ``` ## Failover ### Fail Region In this section, you will simulate a region failure by bringing down the `west` region. 1. Run the following command to stop the Docker containers corresponding to the `west` region: ```bash docker compose stop broker-west-1 broker-west-2 controller-west ``` 2. Verify the new topic replica placement by running the script [describe-topics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/describe-topics.sh): ```bash ./scripts/describe-topics.sh ``` You should see output similar to the following: ```text ==> Describe topic: single-region Topic: single-region PartitionCount: 1 ReplicationFactor: 2 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[]} Topic: single-region Partition: 0 Leader: none Replicas: 2,1 Isr: 1 Offline: 2,1 ==> Describe topic: multi-region-sync Topic: multi-region-sync PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}},{"count":2,"constraints":{"rack":"east"}}],"observers":[]} Topic: multi-region-sync Partition: 0 Leader: 3 Replicas: 1,2,3,4 Isr: 3,4 Offline: 1,2 ==> Describe topic: multi-region-async Topic: multi-region-async PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async Partition: 0 Leader: none Replicas: 2,1,3,4 Isr: 1 Offline: 2,1 Observers: 3,4 ==> Describe topic: multi-region-async-op-under-min-isr Topic: multi-region-async-op-under-min-isr PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=2,confluent.placement.constraints={"observerPromotionPolicy":"under-min-isr","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-under-min-isr Partition: 0 Leader: 4 Replicas: 2,1,4,3 Isr: 4,3 Offline: 2,1 Observers: 4,3 ==> Describe topic: multi-region-async-op-under-replicated Topic: multi-region-async-op-under-replicated PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"observerPromotionPolicy":"under-replicated","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-under-replicated Partition: 0 Leader: 4 Replicas: 1,2,3,4 Isr: 4,3 Offline: 1,2 Observers: 3,4 ==> Describe topic: multi-region-async-op-leader-is-observer Topic: multi-region-async-op-leader-is-observer PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"observerPromotionPolicy":"leader-is-observer","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-leader-is-observer Partition: 0 Leader: none Replicas: 1,2,4,3 Isr: 2 Offline: 1,2 Observers: 4,3 ==> Describe topic: multi-region-default Topic: multi-region-default PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-default Partition: 0 Leader: none Replicas: 2,1,3,4 Isr: 1 Offline: 2,1 Observers: 3,4 ``` 3. After the Confluent Control Center cluster metrics stabilize in about five minutes, you should see output similar to below. ![Control Center fail region](images/c3-fail-region.png) 4. Observe the following: - In the first scenario, the `single-region` topic has no leader, because it had only two replicas in the ISR, both of which were in the `west` region and are now down. - In the second scenario, the `multi-region-sync` topic automatically elected a new leader in `east` (for example, replica 3 in the previous output). Clients can failover to those replicas in the `east` region. - The `multi-region-async`, `multi-region-default` and `multi-region-async-op-leader-is-observer` topics have no leader, because they had only two replicas in the ISR, both of which were in the `west` region and are now down. The observers in the `east` region are not eligible to become leaders automatically because they were not in the ISR. - The `multi-region-async-op-under-min-isr` and `multi-region-async-op-under-replicated` topics have promoted observers into the ISR and an observer has become the leader. This is because their replica placement policy has set `observerPromotionPolicy` to allow this. 5. Run the script [jmx_metrics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/jmx_metrics.sh) to get the JMX metrics for `ReplicasCount`, `InSyncReplicasCount`, `CaughtUpReplicasCount`, and `ObserversInIsrCount` from each of the brokers: ```bash ./scripts/jmx_metrics.sh ``` 6. Verify you see output similar to the following: ```text ==> JMX metric: ReplicasCount single-region: 0 multi-region-sync: 4 multi-region-async: 0 multi-region-async-op-under-min-isr: 4 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 0 multi-region-default: 0 ==> JMX metric: InSyncReplicasCount single-region: 0 multi-region-sync: 2 multi-region-async: 0 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 0 ==> JMX metric: CaughtUpReplicasCount single-region: 0 multi-region-sync: 2 multi-region-async: 0 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 0 ==> JMX metric: ObserversInIsrCount single-region: 0 multi-region-sync: 0 multi-region-async: 0 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 0 ``` ### Failover Observers To explicitly fail over the observers in the `multi-region-async` and `multi-region-default` topics to the `east` region, complete the following steps: 1. Trigger unclean leader election (note: `unclean` leader election may result in data loss): ```bash docker compose exec broker-east-4 kafka-leader-election --bootstrap-server broker-east-4:19094 --election-type UNCLEAN --topic multi-region-async --partition 0 docker compose exec broker-east-4 kafka-leader-election --bootstrap-server broker-east-4:19094 --election-type UNCLEAN --topic multi-region-default --partition 0 ``` 2. Describe the topics again with the script [describe-topics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/describe-topics.sh). ```bash ./scripts/describe-topics.sh ``` You should see output similar to the following: ```text ... ==> Describe topic: multi-region-async Topic: multi-region-async PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async Partition: 0 Leader: 3 Replicas: 2,1,3,4 Isr: 3,4 Offline: 2,1 Observers: 3,4 ... ==> Describe topic: multi-region-default Topic: multi-region-default PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-default Partition: 0 Leader: 3 Replicas: 2,1,3,4 Isr: 3,4 Offline: 2,1 Observers: 3,4 ``` 3. View the changes in the unclean leader election in Confluent Control Center under the “Topics” section. ![Control Center unclean leader election](images/c3-unclean-election.png) 4. Observe the following: - The topics `multi-region-async` and `multi-region-default` have leaders again (for example, replica 3 in the CLI output) - The topics `multi-region-async` and `multi-region-default` had observers that are now in the ISR list (for example, replicas 3,4 in the CLI output) 5. Run the script [jmx_metrics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/jmx_metrics.sh) to get the JMX metrics for `ReplicasCount`, `InSyncReplicasCount`, `CaughtUpReplicasCount`, and `ObserversInIsrCount` from each of the brokers: ```bash ./scripts/jmx_metrics.sh ``` 6. Verify you see output similar to the following: ```text ==> JMX metric: ReplicasCount single-region: 0 multi-region-sync: 4 multi-region-async: 4 multi-region-async-op-under-min-isr: 4 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 0 multi-region-default: 4 ==> JMX metric: InSyncReplicasCount single-region: 0 multi-region-sync: 2 multi-region-async: 2 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 2 ==> JMX metric: CaughtUpReplicasCount single-region: 0 multi-region-sync: 2 multi-region-async: 2 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 2 ==> JMX metric: ObserversInIsrCount single-region: 0 multi-region-sync: 0 multi-region-async: 2 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 2 ``` ### Permanent Failover At this point in the example, if the brokers in the `west` region come back online, the leaders for the `multi-region-async` and `multi-region-default` topics will automatically be elected back to a replica in `west`–that is, replica 1 or 2. This may be desirable in some circumstances, but if you don’t want the leaders to automatically failback to the `west` region, change the topic placement constraints configuration and replica assignment by completing the following steps: 1. For the topic `multi-region-default`, view a modified replica placement policy [placement-multi-region-default-reverse.json](https://github.com/confluentinc/examples/tree/latest/multiregion/config/placement-multi-region-default-reverse.json): ```none { "version": 1, "replicas": [ { "count": 2, "constraints": { "rack": "east" } } ], "observers": [ { "count": 2, "constraints": { "rack": "west" } } ] } ``` 2. Change the replica placement constraints configuration and replica assignment for `multi-region-default`, by running the script [permanent-failover.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/permanent-failover.sh). ```bash ./scripts/permanent-failover.sh ``` The script uses `kafka-configs` to change the replica placement policy and then it runs `confluent-rebalancer` to move the replicas. ```none echo -e "\n==> Switching replica placement constraints for multi-region-default\n" docker compose exec broker-east-3 kafka-configs \ --bootstrap-server broker-east-3:19093 \ --alter \ --topic multi-region-default \ --replica-placement /etc/kafka/demo/placement-multi-region-default-reverse.json echo -e "\n==> Running Confluent Rebalancer on multi-region-default\n" docker compose exec broker-east-3 confluent-rebalancer execute \ --metrics-bootstrap-server broker-ccc:19098 \ --bootstrap-server broker-east-3:19093 \ --replica-placement-only \ --topics multi-region-default \ --force \ --throttle 10000000 docker compose exec broker-east-3 confluent-rebalancer finish \ --bootstrap-server broker-east-3:19093 ``` 3. Describe the topics again with the script [describe-topics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/describe-topics.sh). ```bash ./scripts/describe-topics.sh ``` You should see output similar to the following: ```text ... ==> Describe topic: multi-region-default Topic: multi-region-default PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"east"}}],"observers":[{"count":2,"constraints":{"rack":"west"}}]} Topic: multi-region-async Partition: 0 Leader: 3 Replicas: 3,4,2,1 Isr: 3,4 Offline: 2,1 Observers: 2,1 ... ``` 4. See similar leader placement by clicking on the `multi-region-default` topic and referencing the `Partitions` and `Replica Placement` section. ![Control Center permanent failover](images/c3-permanent-failover-default.png) 5. Observe the following: - For topic `multi-region-default`, replicas 2 and 1, which were previously sync replicas, are now observers and are still offline - For topic `multi-region-default`, replicas 3 and 4, which were previously observers, are now sync replicas. 6. Run the script [jmx_metrics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/jmx_metrics.sh) to get the JMX metrics for `ReplicasCount`, `InSyncReplicasCount`, `CaughtUpReplicasCount`, and `ObserversInIsrCount` from each of the brokers: ```bash ./scripts/jmx_metrics.sh ``` 7. Verify you see output similar to the following: ```text ==> JMX metric: ReplicasCount single-region: 0 multi-region-sync: 4 multi-region-async: 4 multi-region-async-op-under-min-isr: 4 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 0 multi-region-default: 4 ==> JMX metric: InSyncReplicasCount single-region: 0 multi-region-sync: 2 multi-region-async: 2 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 2 ==> JMX metric: CaughtUpReplicasCount single-region: 0 multi-region-sync: 2 multi-region-async: 2 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 2 ==> JMX metric: ObserversInIsrCount single-region: 0 multi-region-sync: 0 multi-region-async: 2 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 0 multi-region-default: 0 ``` ## Failback Now you will bring region `west` back online and restore configuration to the same as in the steady state. 1. Run the following command to bring the `west` region back online: ```bash docker compose start broker-west-1 broker-west-2 controller-west ``` Wait for 5 minutes–the default duration for `leader.imbalance.check.interval.seconds`–until the leadership election restores the preferred replicas. You can also trigger it with `docker compose exec broker-east-4 kafka-leader-election --bootstrap-server \ broker-east-4:19094 --election-type PREFERRED --all-topic-partitions`. 2. Verify the new topic replica placement is restored with the script [describe-topics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/describe-topics.sh). ```bash ./scripts/describe-topics.sh ``` You should see output similar to the following: ```text Topic: single-region PartitionCount: 1 ReplicationFactor: 2 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[]} Topic: single-region Partition: 0 Leader: 2 Replicas: 2,1 Isr: 1,2 Offline: ==> Describe topic: multi-region-sync Topic: multi-region-sync PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}},{"count":2,"constraints":{"rack":"east"}}],"observers":[]} Topic: multi-region-sync Partition: 0 Leader: 1 Replicas: 1,2,3,4 Isr: 3,4,2,1 Offline: ==> Describe topic: multi-region-async Topic: multi-region-async PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,1 Offline: Observers: 3,4 ==> Describe topic: multi-region-async-op-under-min-isr Topic: multi-region-async-op-under-min-isr PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=2,confluent.placement.constraints={"observerPromotionPolicy":"under-min-isr","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-under-min-isr Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 1,2 Offline: Observers: 3,4 ==> Describe topic: multi-region-async-op-under-replicated Topic: multi-region-async-op-under-replicated PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"observerPromotionPolicy":"under-replicated","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-under-replicated Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 1,2 Offline: Observers: 3,4 ==> Describe topic: multi-region-async-op-leader-is-observer Topic: multi-region-async-op-leader-is-observer PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"observerPromotionPolicy":"leader-is-observer","version":2,"replicas":[{"count":2,"constraints":{"rack":"west"}}],"observers":[{"count":2,"constraints":{"rack":"east"}}]} Topic: multi-region-async-op-leader-is-observer Partition: 0 Leader: 2 Replicas: 2,1,3,4 Isr: 2,1 Offline: Observers: 3,4 ==> Describe topic: multi-region-default Topic: multi-region-default PartitionCount: 1 ReplicationFactor: 4 Configs: min.insync.replicas=1,confluent.placement.constraints={"version":1,"replicas":[{"count":2,"constraints":{"rack":"east"}}],"observers":[{"count":2,"constraints":{"rack":"west"}}]} Topic: multi-region-async Partition: 0 Leader: 3 Replicas: 3,4,2,1 Isr: 3,4 Offline: Observers: 2,1 ``` 3. Observe the following: - All topics have leaders again, in particular `single-region` which lost its leader when the `west` region failed. - The leaders for `multi-region-sync` and `multi-region-async` are restored to the `west` region. If they are not, then wait a full 5 minutes (duration of `leader.imbalance.check.interval.seconds`). - The leader for `multi-region-default` stayed in the `east` region because you performed a permanent failover. - Any observers automatically promoted in `multi-region-async-op-under-min-isr` and `multi-region-async-op-under-replicated` are automatically demoted once the `west` region is restored. Leader election is not required for this demotion process, it will happen as soon as the failed region is restored. - The Confluent Control Center topics page is the same as it was at the start of this tutorial. #### NOTE On failback from a failover to observers, any data that wasn’t replicated to observers will be lost because logs are truncated before catching up and joining the ISR. 1. Run the script [jmx_metrics.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/jmx_metrics.sh) to get the JMX metrics for `ReplicasCount`, `InSyncReplicasCount`, `CaughtUpReplicasCount`, and `ObserversInIsrCount` from each of the brokers: ```bash ./scripts/jmx_metrics.sh ``` 2. Verify you see output similar to the following, which should exactly match the output from the start of the tutorial at steady state: ```text ==> JMX metric: ReplicasCount single-region: 2 multi-region-sync: 4 multi-region-async: 4 multi-region-async-op-under-min-isr: 4 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 4 multi-region-default: 4 ==> JMX metric: InSyncReplicasCount single-region: 2 multi-region-sync: 4 multi-region-async: 2 multi-region-async-op-under-min-isr: 2 multi-region-async-op-under-replicated: 2 multi-region-async-op-leader-is-observer: 2 multi-region-default: 2 ==> JMX metric: CaughtUpReplicasCount single-region: 2 multi-region-sync: 4 multi-region-async: 4 multi-region-async-op-under-min-isr: 4 multi-region-async-op-under-replicated: 4 multi-region-async-op-leader-is-observer: 4 multi-region-default: 4 ==> JMX metric: ObserversInIsrCount single-region: 0 multi-region-sync: 0 multi-region-async: 0 multi-region-async-op-under-min-isr: 0 multi-region-async-op-under-replicated: 0 multi-region-async-op-leader-is-observer: 0 multi-region-default: 0 ``` ## Stop the Tutorial and Teardown To stop the example environment and all Docker containers, run the following command: ```bash ./scripts/stop.sh ``` To stop the application manually, run the following command: ```bash docker compose down ``` If you ran either the automated or manual demo to completion, and just want to clean up the Docker environment after the application is stopped, here are some useful commands to do so. - Stop containers: `docker stop $(docker ps -a -q)` - Remove containers: `docker rm $(docker ps -a -q)` - Remove images: `docker rmi $(docker images -q)` (Optional) ## Troubleshooting ### Demo fails on startup The demo application pulls resources from various sites, including OS packages and Docker images. If network connectivity is unreliable or these sites are unavailable, the demo can error out while attempting to pull the needed resources. If this happens, run the following command, and then retry the download and start scripts: ```bash docker compose down ``` ### Containers fail to ping each other If containers fail to ping each other (for example, failures when running the script [validate_connectivity.sh](https://github.com/confluentinc/examples/tree/latest/multiregion/scripts/validate_connectivity.sh)), complete the following steps: 1. Stop the example. ```bash ./scripts/stop.sh ``` 2. Clean up the Docker environment. ```bash docker compose down -v --remove-orphans # More aggressive cleanup docker volume prune ``` 3. Restart the example. ```bash ./scripts/start.sh ``` If the containers still fail to ping each other, restart Docker and run again. ### No detectable latency and jitter If there is no performance difference between the sync replication for the `multi-region-sync` and the other topics, it is possible Docker networking not working or cleaning up properly between runs. 1. Restart Docker. You can restart it via the UI, or: If you are running macOS: ```bash osascript -e 'quit app "Docker"' && open -a Docker ``` If you are running Docker Toolbox: ```bash docker-machine restart ``` ## Additional Resources - [Blog post: Multi-Region Clusters with Confluent Platform 5.4](https://www.confluent.io/blog/multi-region-data-replication)