Cluster Linking Demo (Docker)¶

Looking for Confluent Cloud Cluster Linking docs? You are currently viewing Confluent Platform documentation. If you are looking for Confluent Cloud docs, check out Cluster Linking on Confluent Cloud.

Important

This feature is available as a preview feature. A preview feature is a component of Confluent Platform that is being introduced to gain early feedback from developers. This feature can be used for evaluation and non-production testing purposes or to provide feedback to Confluent.

This is a hands-on demo of Cluster Linking and its capabilities in the context of common use cases. The scripts and README are available on GitHub at demo-scene/cluster-linking

What the Demo Covers¶

By the end of this demo, you will have configured two clusters and successfully used Cluster Linking to migrate data across the clusters for a variety of use cases.

In the following steps, you will:

Create a topic link
Learn how to describe cluster links
Run through a basic topic mirroring example to share topic data
Open up two windows on a command terminal to produce messages on the source cluster and consume from a mirrored topic on the destination cluster
Migrate a consumer from west cluster to east cluster, and learn how to monitor it
Stop linking and perform demo teardown

../../_images/cluster-linking-diagram.png

Setup and Prerequisites¶

The demo includes a Docker Compose file that pulls Docker images and sets up two Kafka clusters, each with its own ZooKeeper:

ZooKeeper
Kafka
Confluent Server

Several scripts are included, which are used to set configurations, run commands, and demo the use cases.

Prerequisites:

Docker
- Docker version 1.11 or later is installed and running.
- Docker Compose is installed. Docker Compose is installed by default with Docker for Mac.
- Docker memory is allocated minimally at 6 GB. When using Docker Desktop for Mac, the default Docker memory allocation is 2 GB. You can change the default allocation to 6 GB in Docker. Navigate to Preferences > Resources > Advanced.
Internet connectivity
Operating System currently supported by Confluent Platform
Networking and Kafka on Docker
- Configure your hosts and ports to allow both internal and external components to the Docker network to communicate.
(Optional) curl.
- In the steps below, you will download a Docker Compose file. You can download this file any way you like, but the instructions below provide the explicit curl command you can use to download the file.

Start the services¶

Clone the Confluent demo-scene repository from GitHub and work in the cluster-linking/ subdirectory, which provides the sample code you will compile and run in this tutorial.

Tip

The following git clone example uses SSH, if your Git configuration is set for HTTPS, use git clone https://github.com/confluentinc/demo-scene.git instead.

git clone git@github.com:confluentinc/demo-scene.git

cd cluster-linking/

Set up a consumer group to read from a topic on the source¶

Open a new command window, create a consumer group in the west cluster that consumes from west-trades.
```
./scripts/6-setup-consumer.sh
```
The set-up-consumer script does the following:
- configures the group with a property to automatically commit offsets
- names the group someGroup
- sets up a consumer to read from the west-trades topic on the west cluster (source) and runs it
You should see output similar to the following as the consumer group reads messages 1-100 from the mirrored topic.
```
==> Consume from west cluster, west-trades and commit offsets (source cluster)

1
2
3
...
99
100
```

In your “lag” window, run the kafka-consumer-groups command to validate offsets on both the source and destination cluster. (These should match.)

To get the offsets for the west cluster (source), provide the following command.

docker-compose exec broker-west kafka-consumer-groups  \
   --bootstrap-server broker-west:19091 \
   --describe \
   --group someGroup

The output for the source cluster should resemble the following.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                               HOST            CLIENT-ID
someGroup       west-trades     0          100             100             0               consumer-someGroup-1-8161b2c7-b9a1-4a81-b2ab-bd58a7a0b2e6 /172.24.0.5     consumer-someGroup-1

To get the offsets for the east cluster (destination), provide the following command.

docker-compose exec broker-east kafka-consumer-groups  \
   --bootstrap-server broker-east:19092 \
   --describe \
   --group someGroup

The output for the destination cluster should resemble the following.

Consumer group 'someGroup' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
someGroup       west-trades     0          100             100             0               -               -               -

Tip

It can take up to 10 seconds for offsets to be migrated.

Migrate a group from source to destination¶

Update the offset migration to stop migrating consumer offsets from the west cluster to the east cluster.

The migrate-one-cg script updates the offset migration and cluster link to accomplish the migration, produces more messages to the west-trades topic, and then consumes them from the consumer group in the east.

./scripts/7-migrate-one-cg.sh

Watch the output messages and note that the script accomplishes the following tasks.

Updates consumer.offset.group.filters to set an exclusion filter for someGroup which excludes it from consumer offsets
Uses the kafka-configs command to update the cluster link from the east to exclude migration of someGroup consumer offsets
Produces another 100 messages in the west cluster
Consumes the new consumer in the east cluster
Monitors the consumer offsets that have been migrated from one side to the other

==> Stop migrating the consumer group someGroup via the west link
Completed updating config for cluster-link west-cluster-link.
==> Produce 100 more messages to the source topic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
==> Consume from east cluster, west-trades and commit offsets (destination cluster)
101
102
...
197
198
199
200
^CProcessed a total of 100 messages
==> Monitor that the consumer offsets have correctly been migrated
==> West Cluster
Consumer group 'someGroup' has no active members.
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
someGroup       west-trades     0          100             200             100             -               -               -
==> East Cluster
Consumer group 'someGroup' has no active members.
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
someGroup       west-trades     0          200             200             0               -               -               -

In your “lag” window, rerun the kafka-consumer-groups commands on both clusters to verify that the migrated consumer group on broker-east is fully caught up to offset 200.

First, on the west cluster. (This is the cluster you produced to, so just verify that the producer worked.)

docker-compose exec broker-west kafka-consumer-groups  \
   --bootstrap-server broker-west:19091 \
   --describe \
   --group someGroup

The output for the source cluster should resemble the following.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                               HOST            CLIENT-ID
someGroup       west-trades     0          200             200             0               consumer-someGroup-1-8161b2c7-b9a1-4a81-b2ab-bd58a7a0b2e6 /172.24.0.5     consumer-someGroup-1

Now, verify that broker-east is caught up.

docker-compose exec broker-east kafka-consumer-groups  \
   --bootstrap-server broker-east:19092 \
   --describe \
   --group someGroup

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                               HOST            CLIENT-ID
someGroup       west-trades     0          200             200             0               consumer-someGroup-1-dff88343-140b-481b-acb2-e86d0713e180 /172.24.0.4     consumer-someGroup-1

These should match, with no lag because we are not producing more messages at this point.

Change a topic from a mirrored topic to a writable topic¶

Run the stop-link script to change a topic from a mirrored topic to writable topic.

The script uses kafka-replica-status to show the mirrored topic, kafka-topics --alter --mirror-action stop to stop the link, and kafka-replica-status again to monitor the changes.

./scripts/8-stop-link.sh

You should see output similar to the following as the link stops, and the mirrored topic changes to a writable topic.

==> Using replica status to see mirrored topic

Topic              Partition Replica ClusterLink       IsLeader IsObserver IsIsrEligible IsInIsr IsCaughtUp LastCaughtUpLagMs LastFetchLagMs LogStartOffset LogEndOffset
 __consumer_offsets 0         1       -                 true     false      true          true    true       0                 0              0              0
 __consumer_offsets 1         1       -                 true     false      true          true    true       0                 0              0              0
 ...
 __consumer_offsets 49        1       -                 true     false      true          true    true       0                 0              0              0
 _confluent-license 0         1       -                 true     false      true          true    true       0                 0              0              0
 west-trades        0         1       -                 true     false      true          true    true       0                 0              0              500
 west-trades        0         1       west-cluster-link true     false      true          true    true       -7                -7             0              500

 ==> Stop west-link
 Topic 'west-trades's mirror was successfully stopped.

 ==> Monitor the change in mirrored topic status
 Topic              Partition Replica ClusterLink IsLeader IsObserver IsIsrEligible IsInIsr IsCaughtUp LastCaughtUpLagMs LastFetchLagMs LogStartOffset LogEndOffset
 __consumer_offsets 0         1       -           true     false      true          true    true       0                 0              0              0
 __consumer_offsets 1         1       -           true     false      true          true    true       0                 0              0              0
 ...
 __consumer_offsets 49        1       -           true     false      true          true    true       0                 0              0              0
 _confluent-license 0         1       -           true     false      true          true    true       0                 0              0              0
 west-trades        0         1       -           true     false      true          true    true       0                 0              0              500

Teardown¶

Shut down any running producers or consumers cleanly with Ctl-C in their respective command windows. (If you forget, the shutdown script also will stop these for you.)

Run the shutdown script to stop and remove Docker containers.

./scripts/9-shutdown.sh

Your output should resemble:

Error response from daemon: No such container: pumba-latency
Stopping broker-west    ... done
Stopping broker-east    ... done
Stopping zookeeper-west ... done
Stopping zookeeper-east ... done
Removing broker-west    ... done
Removing broker-east    ... done
Removing zookeeper-west ... done
Removing zookeeper-east ... done
Removing network clusterlinking_n1

For more information, refer to the official Docker documentation.