Cluster Linking Active-Passive Manual Switchover Tutorial

This tutorial shows you how to manually fail over and fail back an active-passive Confluent Cloud cluster pair using Cluster Linking.

Your primary cluster in the active region processes all production read and write traffic during normal operations, while the secondary cluster in the DR region maintains read-only mirror topics. This manual process is for deployments that don’t use Automatic Failover.

When a regional disaster occurs, you manually stop production workloads on the primary cluster, promote the mirror topics on the DR cluster to standard read-write topics, and redirect your application clients to the DR cluster bootstrap endpoint.

Prerequisites and considerations

Before setting up an active-passive manual switchover architecture, review the following prerequisites and constraints.

Cluster and account prerequisites

• Supported cluster types: Dedicated or Enterprise clusters only on both primary and secondary clusters (Basic, Standard, and Freight are not supported).

• Environment and Organization scope: Both primary and DR clusters must be in the same Confluent Cloud organization. They can be in the same environment or in different environments, with tradeoffs to consider with either setup.

• Cluster sizing: Size both clusters identically, the same Confluent Unit for Kafka (CKU) count, so the secondary cluster can absorb full primary traffic upon failover.

• Bidirectional cluster link: As a best practice, create the cluster link in bidirectional mode from the outset. This enables direction flipping during failback and allows critical metadata sync.

Client prerequisites

• Dynamic client re-bootstrapping: Client applications must be designed to update their bootstrap server configuration (for example, HashiCorp Consul, Vault, or AWS Secrets Manager) and restart during a failover event.

• Pre-provisioned API keys and credentials: Create API keys and secrets on the secondary (DR) cluster ahead of time. Store these in your key manager so clients can authenticate immediately upon failover.

Networking and connectivity prerequisites

Network connectivity: Before failover, configure network paths (virtual private cloud (VPC) or virtual network (VNet) peering, PrivateLink, or Transit Gateways) between your applications and both the primary and secondary clusters.

Schema and governance constraints

• Schema validation properties: Topic-level schema validation properties (confluent.key.schema.validation and confluent.value.schema.validation) are not automatically replicated by Cluster Linking. You must manually configure these settings on the DR cluster.

• Schema Registry DR: Schema Registry failover is managed separately from Apache Kafka® topic failover. Ensure Schema Linking is configured between your primary and secondary Schema Registry clusters.

What the tutorial covers

This tutorial demos use of the Confluent CLI Cluster Linking commands to create a DR cluster and fail over to it.

You start by building a cluster link to the DR cluster (destination) and mirroring all pertinent topics, access control lists (ACLs), and consumer group offsets. This is the “steady state” setup.

Then you incur a sample outage on the original (source) cluster. When this happens, the producers, consumers, and cluster link cannot interact with the original cluster.

You then call a failover command that converts the mirror topics on the DR cluster into regular topics.

Finally, you move producers and consumers over to the DR cluster, and continue operations. The DR cluster has become the new source of truth.

Set up steady state

Create or choose the clusters you want to use

1. Log in to the Confluent Cloud Console.

2. If you do not already have clusters created, create two clusters as described in Create a Kafka Cluster.

   Both clusters must be a Dedicated or Enterprise cluster (Basic, Standard, and Freight clusters are not supported).

   Name these original and dr.

   

Tradeoffs between using the same or different environments

You can create the active cluster and DR cluster in either the same environment or in different environments. Each approach has tradeoffs.

Same environment

Following are advantages and disadvantages of placing the active cluster and DR cluster in the same environment.

Advantages

• Simplified management with a single environment to administer.

• Easier setup for Cluster Linking, API keys, and permissions.

• Simpler cost tracking and billing within one environment.

• Streamlined ACL and role-based access control (RBAC) management for security purposes.

Disadvantages

• Both clusters could be affected by environment-level issues or misconfigurations.

• Less organizational separation between production and DR resources.

• Might be less suitable for multi-team scenarios where different teams manage different environments.

Different environments

Following are advantages and disadvantages of placing the active cluster and DR cluster in different environments.

Advantages

• Better isolation and resilience to environment-level issues.

• Stronger organizational separation: useful when different teams or departments manage production and DR.

• More granular access controls at the environment level.

• Reduced risk of accidental changes affecting both clusters.

Disadvantages

• More complex management across multiple environments.

• Additional overhead for managing API keys, service accounts, and permissions across environments.

• Requires careful coordination of Cluster Linking configuration between environments.

Navigate to the environment and clusters

Log on to the CLI (confluent login), and make sure you are in the environment you want to use for this tutorial.

To view environments and the clusters they contain, use these commands.

Command	Description
List environments	confluent environment list
Select an environment.	confluent environment use <environment-id>
List clusters.	confluent kafka cluster list
Select a cluster.	confluent kafka cluster use <cluster-id>

The tutorial provides a few reminders about this navigation, but remember you can use this when you use commands without specifying a resource. For example, confluent kafka link list or confluent kafka topic list gives you the cluster links and topics, respectively, for the currently selected cluster. Otherwise, specify the resource. For example, confluent kafka link list --cluster <cluster-id> or confluent kafka topic list --cluster <cluster-id>.

Also, keep in mind, as you walk through these steps, that you can view most cluster and topic information on the Cloud Console at the Confluent Cloud Console. You do all the work on the CLI, but you might want to use the console as a supplemental view.

Keep notes of required information

This tutorial requires several pieces of information from the Confluent CLI that you use throughout the steps.

Tip

To keep track of this information, you might find it easiest to save the output of the Confluent CLI commands to a text file and use shell environment variables. If you do so, be sure to safeguard API keys and secrets afterwards by deleting files, or moving only the security codes to safer storage. For details on how to do this, and other time-savers, see CLI tips in the overview.

For this walkthrough, you need the following details accessible through Confluent CLI commands.

• The cluster ID of your original cluster (<original_cluster_id> for purposes of this tutorial). To get cluster IDs on Confluent Cloud, type the following at the Confluent CLI. In the following examples, the original cluster ID is lkc-xkd1g.

  confluent kafka cluster list
  

• The bootstrap server for the original cluster. To get this, type the following command, replacing <original_cluster_id> with the cluster ID for your source cluster.

  confluent kafka cluster describe <original_cluster_id>
  

  Your output should resemble the following. In the following example output, the bootstrap server is the value: SASL_SSL://pkc-9kyp5.us-east-1.aws.confluent.cloud:9092. This value is referred to as <original-bootstrap-server> in this tutorial.

  +--------------+---------------------------------------------------------+
  | Id           | lkc-xkd1g                                               |
  | Name         | AWS US                                                  |
  | Type         | DEDICATED                                               |
  | Ingress      |                                                      50 |
  | Egress       |                                                     150 |
  | Storage      | Infinite                                                |
  | Cloud        | aws                                                     |
  | Availability | single-zone                                             |
  | Region       | us-east-1                                               |
  | Status       | UP                                                      |
  | Endpoint     | SASL_SSL://pkc-9kyp5.us-east-1.aws.confluent.cloud:9092 |
  | ApiEndpoint  | https://pkac-rrwjk.us-east-1.aws.confluent.cloud        |
  | RestEndpoint | https://pkc-9kyp5.us-east-1.aws.confluent.cloud:443     |
  | ClusterSize  |                                                       1 |
  +--------------+---------------------------------------------------------+
  

• The cluster ID of your DR cluster (<dr_cluster_id>). You can get this in the same way as your original cluster ID, with this command confluent kafka cluster list.

  confluent kafka cluster list
  

• The bootstrap server of your DR cluster (<dr_bootstrap_server>). You can get this in the same way as your original cluster bootstrap server, by using the describe command, replacing <dr_cluster_id> with your destination cluster ID.

  confluent kafka cluster describe <dr_cluster_id>
  

Create a cluster link between the original and the DR cluster

Start by creating a cluster link that is mirroring topics, ACLs, and consumer group offsets from the source cluster to the destination cluster.

Set up privileges for the cluster link

Your cluster link needs privileges to read the appropriate topics on your source cluster. To give it these privileges, you create two mechanisms:

• A service account for the cluster link. Service accounts are used in Confluent Cloud to group together applications and entities that need access to your Confluent Cloud resources.

• An API key and secret that is associated with the cluster link’s service account and the source cluster. The link uses this API key to authenticate with the source cluster when it is fetching topic information and messages. A service account can have many API keys, but for this tutorial, you need only one.

To create these resources, do the following:

1. Create a service account for this cluster link.

   confluent iam service-account create Cluster-Linking-Demo --description "For the cluster link created for the DR failover tutorial"
   

   Your output should resemble the following.

   +-------------+-----------+
   | Id          |    254122 |
   | Resource ID | sa-lqxn16 |
   | Name        | ...       |
   | Description | ...       |
   +-------------+-----------+
   

   Save the ID field (<service_account_id> for the purposes of this tutorial).

2. Create the API key and secret.

   confluent api-key create --resource <original_cluster_id> --service-account <service_account_id>
   

   Note

   Store this key and secret somewhere safe. When you create the cluster link, you must supply it with this API key and secret, which is stored on the cluster link itself.

3. Allow the cluster link to read topics on the source cluster. Give the cluster link’s service account the ACLs to READ and DESCRIBE_CONFIGS for all topics.

   confluent kafka acl create --allow --service-account <service_account_id> --operations read,describe-configs --topic "*" --cluster <original_cluster_id>
   

   The preceding example allows read access to all topics by using the asterisk (--topic "*") instead of specifying particular topics. If you wish, you can narrow this down to a specific set of topics to mirror. For example, to allow the cluster link to read all topics that begin with a “clicks” prefix, you can do this:

   confluent kafka acl create --allow --service-account <service_account_id> --operations read,describe-configs --topic clicks --prefix --cluster <original_cluster_id>
   

4. Provide the capability to sync ACLs from the source to the destination cluster.

   This allows consumers, producers, and other services to continue running, even in the event of a failure.

   To do this, you must give the cluster link’s service account an ACL to DESCRIBE the source cluster:

   confluent kafka acl create --allow --service-account <service_account_id> --operations describe --cluster-scope --cluster <original_cluster_id>
   

5. Provide the capability to sync consumer group offsets for mirror topics over this cluster link, so that consumers can pick up at the offset at which they left off.

   Your clusters need two sets of ACLs for this.

   • Give the cluster link’s service account the appropriate ACLs to DESCRIBE topics, and to READ and DESCRIBE consumer groups on the source (original) cluster.

     confluent kafka acl create --allow --service-account <service_account_id> --operations describe --topic "*" --cluster <original_cluster_id>
     

     confluent kafka acl create --allow --service-account <service_account_id> --operations read,describe --consumer-group "*" --cluster <original_cluster_id>
     

   • Give the cluster link’s service account ACLs to READ and ALTER topics on the destination (DR) cluster, and ACLs to READ its consumer groups.

     confluent kafka acl create --allow --service-account <service_account_id> --operations read,alter --topic "*" --cluster <dr_cluster_id>
     

     confluent kafka acl create --allow --service-account <service_account_id> --operations read --consumer-group "*" --cluster <dr_cluster_id>
     

Create the cluster link

1. Create a configuration file that turns on ACL sync, consumer group offset sync, and includes the security credentials for the link.

   To do this, copy the following lines into a new file called dr-link.config, and then replace <api_key> and <api_secret> with the key and secret you just created.

   consumer.offset.sync.enable=true
   consumer.offset.group.filters={"groupFilters": [{"name": "*","patternType": "LITERAL","filterType": "INCLUDE"}]}
   consumer.offset.sync.ms=1000
   
   acl.sync.enable=true
   acl.sync.ms=1000
   acl.filters={ "aclFilters": [ { "resourceFilter": { "resourceType": "any", "patternType": "any" }, "accessFilter": { "operation": "any", "permissionType": "any" } } ] }
   
   topic.config.sync.ms=1000
   
   link.mode=BIDIRECTIONAL
   
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<api_key>" password="<api_secret>";
   

   A few notes on this configuration, which does the following:

   • Syncs offsets for all consumer groups (*) for these mirror topics. You can filter these down by passing in either exact topic names ("patternType": "LITERAL") or prefixes ("patternType": "PREFIX") to either include ("filterType": "INCLUDE") or exclude ("filterType": "EXCLUDE").

   • Syncs all ACLs that are on the original cluster (*) so that consumers, producers, and other services can access the topics on the DR cluster. Unlike the consumer group offset sync, this is not limited to mirror topics, and might sync ACLs for other topics. You can filter these down by passing in either exact topic names ("patternType": "LITERAL") or prefixes ("patternType": "PREFIX") to either include ("filterType": "INCLUDE") or exclude ("filterType": "EXCLUDE").

   • Syncs consumer offsets, ACLs, and topic configurations every 1000 milliseconds. This gives the link more up-to-date values for these. It comes at the expense of higher data throughput over the link, and potentially lower maximum throughput for the topic data. As a best practice, try different values to see what works best for your specific cluster link.

   • The last line in the file starts with sasl.jaas.config and ends with a semicolon (;). This must be all on one line, as shown.

2. Create the cluster link as shown in the following example, with the command confluent kafka link create <flags>.

   In this example, the cluster link is called dr-link.

   confluent kafka link create dr-link \
     --cluster <dr_cluster_id>  \
     --source-cluster <original_cluster_id> \
     --source-bootstrap-server <original-bootstrap-server>  \
     --config dr-link.config
   

   If this is successful, you should get this message: Created cluster link "dr-link".

   Also, you can verify that the link was created by listing existing links:

   confluent kafka link list --cluster <dr_cluster_id>
   

   Tip

   --source-cluster-id was replaced with --source-cluster in version 3 of confluent CLI, as described in the command reference for confluent kafka link create.

Configure the source and mirror topics

1. Create a topic called dr-topic on the original cluster.

   For the sake of this demo, create this topic with only one partition (--partitions 1). Having only one partition makes it easier to notice how the consumer offsets are synced from original cluster to DR cluster.

   confluent kafka topic create dr-topic --partitions 1 --cluster <original_cluster_id>
   

   You should see the message Created topic "dr-topic", as shown in the following example.

   confluent kafka topic create dr-topic --partitions 1 --cluster lkc-xkd1g
   

   Your output should resemble the following.

   Created topic "dr-topic".
   

   You can verify this by listing topics.

   confluent kafka topic list --cluster <original_cluster_id>
   

2. Create a mirror topic of dr-topic on the DR cluster.

   confluent kafka mirror create dr-topic --link <link-name> --cluster <dr_cluster_id>
   

   You should see the message Created mirror topic "dr-topic", as shown in the following example.

   confluent kafka mirror create dr-topic --link dr-link --cluster lkc-r68yp
   

   Your output should resemble the following.

   Created mirror topic "dr-topic".
   

At this point, you have a topic on your original cluster that is mirroring all of its data, ACLs, and consumer group offsets to a mirror topic on your DR cluster.

Produce and consume some data on the original cluster

In this section, you simulate an application that is producing data to and consuming data from your original cluster. You use the Confluent CLI produce and consume commands to do so.

1. Create a service account to represent your CLI based demo clients.

   confluent iam service-account create CLI --description "From CLI"
   

   Your output should resemble:

   +-------------+-----------+
   | Id          |    254262 |
   | Resource ID | sa-ldr3w1 |
   | Name        | CLI       |
   | Description | From CLI  |
   +-------------+-----------+
   

   In the preceding example, the <cli_service-account_id> is sa-ldr3w1.

2. Create an API key and secret for this service account on your original cluster, and save these as <original_cli_api_key> and <original_cli_api_secret>.

   confluent api-key create --resource <original_cluster_id> --service-account <cli_service-account_id>
   

3. Create an API key and secret for this service account on your DR cluster, and save these as <DR_CLI_api_key> and <DR_CLI_api_secret>.

   confluent api-key create --resource <dr_cluster_id> --service-account <cli_service-account_id>
   

4. Give your CLI service account enough ACLs to produce and consume messages on your original cluster.

   confluent kafka acl create --service-account <cli_service-account_id> --allow --operations read,describe,write --topic "*" --cluster <original_cluster_id>
   

   confluent kafka acl create --service-account <cli_service-account_id> --allow --operations describe,read --consumer-group "*" --cluster <original_cluster_id>
   

Now you can produce and consume some data on the original cluster.

1. Tell your CLI to use your original API key on your original cluster.

   confluent api-key use <original_cli_api_key> --resource <original_cluster_id>
   

2. Produce the numbers 1-5 to your topic.

   seq 1 5 | confluent kafka topic produce dr-topic --cluster <original_cluster_id>
   

   You should see this output, but the command should complete without needing you to press ^C or ^D.

   Starting Kafka Producer. ^C or ^D to exit
   

   Tip

   If you get an error message indicating unable to connect to Kafka cluster, wait for a minute or two, then try again. For recently created Kafka clusters and API keys, it might take a few minutes before the resources are ready.

3. Start a CLI consumer to read from the dr-topic topic, and give it the name cli-consumer.

   As a part of this command, pass in the flag --from-beginning to tell the consumer to start from offset 0.

   confluent kafka topic consume dr-topic --group cli-consumer --from-beginning
   

   After the consumer reads all five messages, press Ctrl+C to stop the consumer.

4. To observe how consumers pick up from the correct offset on a failover, artificially force some consumer lag on your consumer.

   Produce numbers 6-10 to your topic.

   seq 6 10 | confluent kafka topic produce dr-topic
   

   You should see the following output, but the command should complete without needing you to press ^C or ^D.

   Starting Kafka Producer. ^C or ^D to exit
   

Now, you have produced 10 messages to your topic on your original cluster, but your cli-consumer has consumed only five.

Monitor mirroring lag

Because Cluster Linking is an asynchronous process, there might be mirroring lag between the source cluster and the destination cluster.

You can see what your mirroring lag is on a per-partition basis for your DR topic with this command:

confluent kafka mirror describe dr-topic --link dr-link --cluster <dr_cluster_id>

  LinkName  | MirrorTopicName | Partition | PartitionMirrorLag | SourceTopicName | MirrorStatus | StatusTimeMs
+-----------+-----------------+-----------+--------------------+-----------------+--------------+---------------+
   dr-link |  dr-topic        |         0 |                  0 |  dr-topic       | ACTIVE       | 1624030963587

You can also monitor your lag and your mirroring metrics through the Confluent Cloud Metrics. These two metrics are exposed:

• MaxLag shows the maximum lag (in number of messages) among the partitions that are being mirrored. It is available on a per-topic and a per-link basis. This gives you a sense of how much data is on the original cluster only at the point of failover.

• Mirroring Throughput shows on a per-link or per-topic basis how much data is being mirrored.

Active-passive failover and failback process

An active-passive DR plan is designed for cases where you have producers only on one side. In these cases, you can have consumer groups on both sides, as long as they use different consumer group names.

Cluster Linking enables fast failover to the DR region and fast failback to the original region, so you can test your DR workflow with minimal risk.

Using the Cluster Linking truncate-and-restore command, a DR workflow can be completed with:

• Minimal downtime when performing the failover or failback operations.

• Option to fail back as soon as the failover is complete, or to stay running in the DR region for as long as desired, with no added cost from Confluent Cloud.

Fail over

When a disaster strikes, you can use the failover command to quickly and safely fail-forward to your secondary region so your clients can continue processing data and reduce downtime as much as possible.

After the disaster is over, you can reinstate the original primary region to ensure redundancy is built into your systems. The truncate-and-restore command makes this possible. After an outage has occurred, and the primary cluster is back up and running, run truncate-and-restore on the primary topics.

During this process, the primary and secondary clusters and topics essentially switch places. The original primary topics become mirror topics and start fetching from the original secondary topics (treating them now as primary). This reinstates the secondary region to ensure redundancy.

Note

The truncate-and-restore process truncates any divergent records written after the failover point on the primary topic and starts fetching records from the other cluster to become a read-only mirror topic, so it’s important to note that any records that were written to the original primary cluster after the failover point are lost with the use of this command.

Fail back

After failover, you can go back to the original steady state (“fail back”) if desired.

To fail back, run the reverse-and-start command on the new mirror topic on the original primary cluster, so that the original primary cluster becomes the writable topic, and the original secondary cluster becomes the mirror topic. Note that you must also shut down your clients and restart them at your original primary cluster.

List cluster links and mirror topics

At various points in a workflow, it may be useful to get lists of cluster linking resources such as links or mirror topics. You might want to do this for the purposes of monitoring, or before starting a failover or migration.

To list the cluster links on the active cluster:

confluent kafka link list

You can get a list mirror topics on a link, or on a cluster.

To list the mirror topics on a specified cluster link:

confluent kafka mirror list --link <link-name> --cluster <cluster-id>

To list all mirror topics on a particular cluster:

confluent kafka mirror list --cluster <cluster-id>

Simulate a failover to the DR cluster

In a disaster event, your original cluster is usually unreachable.

In this section, you go through the steps you follow on the DR cluster to resume operations.

1. Perform a dry run of a failover to preview the results without actually executing the command. To do this, add the --dry-run flag to the end of the command.

   confluent kafka mirror failover <mirror-topic-name> --link <link-name> --cluster <dr_cluster_id> --dry-run
   

   For example:

   confluent kafka mirror failover dr-topic --link dr-link --cluster <dr_cluster_id> --dry-run
   

2. Stop the mirror topic to convert it to a normal, writable topic.

   confluent kafka mirror failover <mirror-topic-name> --link <link-name> --cluster <dr_cluster_id>
   

   For this example, the mirror topic name and link name are as follows.

   confluent kafka mirror failover dr-topic --link dr-link --cluster <dr_cluster_id>
   

   Expected output:

   MirrorTopicName | Partition | PartitionMirrorLag | ErrorMessage | ErrorCode
   -------------------------------------------------------------------------
   dr-topic        |         0 |                  0 |              |
   

   The failover command is irreversible and converts the mirror topic to a regular topic. To restore mirroring, you can use the truncate-and-restore command on the original source topic. For the full restore and failback procedure, see Active-passive failover and failback process.

3. Now you can produce and consume data on the DR cluster.

   Set your CLI to use the DR cluster’s API key:

   confluent api-key use <DR-CLI-api-key> --resource <dr_cluster_id>
   

4. Produce numbers 11-15 on the topic, to show that it is a writable topic.

   seq 11 15 | confluent kafka topic produce dr-topic --cluster <dr_cluster_id>
   

   You should see this output, but the command should complete without needing you to press ^C or ^D.

   Starting Kafka Producer. ^C or ^D to exit
   

5. Move your consumer group to the DR cluster, and consume from dr-topic on the DR cluster.

   confluent kafka topic consume dr-topic --group cli-consumer --cluster <dr_cluster_id>
   

   You should expect the consumer to start consuming at number 6, because that’s where it left off on the original cluster. If it does, that shows that its consumer offset was correctly synced. It should consume through number 15, which is the last message you produced on the DR cluster.

   After you see number 15, press Ctrl+C to stop the consumer.

   Expected output:

   Starting Kafka Consumer. ^C or ^D to exit
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   ^CStopping Consumer.
   

You have now failed over your CLI producer and consumer to the DR cluster, where they continued operations smoothly.

Related content

• Learn more about Mirror Topics concepts.

• Review Cluster Linking Metrics and Lag Monitoring.

• Compare this procedure with Active-Passive Automatic Failover.