Verify Replicator Configuration for Confluent Platform

The Replicator verifier is a command line tool for identifying issues with a Replicator configuration and recommending remedial actions. This tool accepts a connector or an executable configuration and runs a number of checks against the clusters referenced. This can be useful for:

  • Preparing new deployments - the verifier tool provides confidence in a new Replicator flow configuration without actually deploying the Replicator
  • Diagnosing Replicator issues in existing flows - the verifier tool can quickly identify the area of the issue without redeploying the flow

Tip

The Replicator verifier is available as part of a Confluent Platform installation, included in the $CONFLUENT_HOME/bin directory. Once you have Confluent Platform installed and up-and-running, type replicator-verifier at the command line to get a read-out on the command and its options.

Running the verifier

Run the Replicator verifier using the configuration from three sources:

Note

Replicator verifier should be run from the same host that will run Replicator

  • From Replicator executable:

    replicator-verifier \
 --consumer.config ./consumer.properties \
 --producer.config ./producer.properties \
 --replication.config ./replication.properties

  • From a connector json:

    replicator-verifier \
 --replicator-json ./replicator.json

  • From a deployed connector:

    replicator-verifier \
 --connect-url localhost:8083 \
 --connector-name replicator

On completion the Replicator verifier will produce a summary of checks performed and their results:

[2020-02-28 12:39:02,252] INFO All checks completed. (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,252] INFO              Source cluster describe configs check : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO                     Connectivity check Destination : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO              topic.whitelist/topic.blacklist check : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO                              License manager check : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO                Consumer offset commit check source : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO           Consumer offset commit check destination : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO                        Destination topic ACL check : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO                                  Topic regex check : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO                             Source topic ACL check : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO                          Connectivity check Source : PASSED (io.confluent.connect.replicator.Verifier)
[2020-02-28 12:39:02,253] INFO Verifier run complete. (io.confluent.connect.replicator.Verifier)

Any failed checks are accompanied by guidance explaining the relevant parameters to resolve the issue:

[2020-02-28 12:41:46,846] INFO                              License manager check : FAILED
By default, Replicator will assume the license is stored in the _confluent-command topic on the destination cluster.
Ensure that the properties prefixed "dest.kafka." refer to your license cluster, topic and principal.
Alternatively, these can be overridden using the "confluent.topic." prefix (e.g. confluent.topic.bootstrap.servers).
(io.confluent.connect.replicator.Verifier)

Important

Replicator verifier does not currently support monitoring interceptors. Configurations like *.interceptor.classes should be removed from Replicator configurations before running the verifier.

Command Line Parameters of Replicator Verifier

The available command line parameters are:

Command line parameter Value Description
--connect-url <Connect worker url> URL for the Connect instance running connectors
--connector-name <Replicator Connector name> Name for the replicator connector
--replicator-json <replicator.json> JSON file containing configuration for replicator running as a connector
--producer.config <producer.properties> Producer config file for replicator running in executable mode
--consumer.config <consumer.properties> Consumer config file for replicator running in executable mode
--replication.config <replication.properties> Replicator config file for replicator running in executable mode
--destination-test-topic <topic name> Name of destination topic, which is an override for the randomized topic name used for dry run checks at the destination cluster (this topic must already exist)

Checks performed

Replicator verifier performs the following checks to determine that the configuration provided is valid.

Non-intrusive checks

  • Source cluster connectivity check - Confirms Replicator can connect to the source cluster.
  • Destination cluster connectivity check - Confirms Replicator can connect to the destination cluster.
  • Topic whitelist/blacklist check - Confirms the list of topics to be whitelisted/blacklisted are valid and not conflicting.
  • Topic regex check - Confirms any regex topic matching pattern is valid and displays they set of topics matched.
  • License check - Confirms that a valid license to use Replicator is present.
  • Source topic read checks - Confirms that Replicator can read the configured source topics.
  • Source topic describe check - Confirms that Replicator can describe the configured source topics.
  • Consumer timestamps test - Confirms Replicator can read/write timestamps to the internal timestamps topic for offset translation.

Intrusive checks

Note

Intrusive checks will temporarily change the state of the cluster they are performed on. These changes take the form of topic creation/manipulation and consumer offset commits. Replicator verifier will prompt for approval before running these checks and as such should be run in interactive mode.

  • Destination topic check - Confirms that Replicator can perform the following required actions on a sample destination topic:
    1. Create
    2. Describe
    3. Describe Configs
    4. Alter
    5. Produce
  • Source cluster offset commit - Confirms that Replicator can commit consumer offsets to the source cluster
  • Destination cluster offset commit - Confirms that Replicator can commit consumer offsets to the destination cluster