Troubleshoot Cluster Linking on Confluent Platform¶
This page describes common errors you may encounter when creating cluster links, and how to address them. Task error codes for cluster link and mirror topic commands are also listed.
Errors when Creating a Cluster Link¶
The command to create a cluster link can fail with one of the following error codes. A customized error message will also be returned, to provide specific context about the clusters being used.
UNRESOLVABLE_BOOTSTRAP_ERROR¶
Meaning: The destination cluster (or, for a source-initiated link, the source cluster) could not resolve the provided bootstrap server to an IP address. In other words, there was no DNS entry for this bootstrap server in the cluster’s DNS.
Solutions:
- If the provided source cluster bootstrap server is a URL with a FQDN that is not Confluent Cloud
(for example,
kafka.example.com:9092
), then the DNS is not properly configured for the destination cluster to resolve that FQDN to an IP address. - If the destination cluster is a Confluent Cloud cluster, this error is likely because your FQDN relies
on your private DNS; there is no way to provide Confluent Cloud with access to your private DNS. In this
case, you have two options:
- The best way to resolve this error is to use a FQDN that is registered in a public DNS.
Alternatively, use an IP address (such as,
10.10.4.24:9092
) for the bootstrap server (instead of a host name). Your source cluster must have an IP address advertised listener on every broker before you can use Cluster Linking with an IP address. - Alternatively, you can register this FQDN with a public DNS, pointing to the appropriate IP address of your broker, proxy, or load balancer (internal IPs that can be registered in a public DNS), so that Confluent Cloud can resolve this FQDN to an IP that it can reach.
- The best way to resolve this error is to use a FQDN that is registered in a public DNS.
Alternatively, use an IP address (such as,
- If the destination cluster is a Confluent Platform cluster, then you can attempt to fix the destination cluster’s DNS.
Ensure the route to your source cluster allows traffic over port
9092
. - If the provided source cluster bootstrap server is for Confluent Cloud (for example,
pkc-lmn0p.us-west-2.aws.confluent.cloud:9092
), then contact Confluent Support.
BOOTSTRAP_TCP_CONNECTION_FAILED_ERROR¶
Meaning: The destination cluster cannot reach a Apache Kafka® cluster at the provided source bootstrap server.
Solutions:
If both clusters are Confluent Cloud clusters:
- First verify that your source cluster is up and running, and in a healthy state. (If it isn’t, this will cause the error.)
- If the source cluster has Internet networking, contact Confluent Support.
- If the source cluster has Transit Gateway networking, follow the steps for troubleshoot Transit Gateway connectivity
- If the source cluster does not have Internet networking or Transit Gateway networking, make sure that the two Confluent Cloud clusters exist in the same Confluent Cloud network, region, and cloud provider. Cluster Linking is not supported between private Confluent Cloud clusters in different Confluent Cloud networks, other than with Transit Gateway.
If the source cluster is a Confluent Platform or Kafka cluster, and the destination cluster is a Confluent Cloud cluster: Your Confluent Cloud cluster cannot reach your source cluster and you must resolve this. Providing network access for Confluent Cloud to reach your source cluster is your responsibility.
- If you are intending to access a source cluster over the internet, validate that all of your source cluster’s
brokers have public internet IP addresses. You can verify this by consuming from your cluster (for example, with
kafka-console-consumer
) on a machine that does not have VPN or other private access to your cluster. For the consumer configuration, enter the bootstrap servers and security configuration that you provided to the cluster link. - If you are intending to access a source cluster over private internet–that is, using private IP addresses–this error indicates
that you have not properly set up the networking between Confluent Cloud and the source cluster. Double check that your networking meets these requirements:
- The Confluent Cloud cluster must have a networking type that is either VPC Peered, VNet Peered, or Transit Gateway.
- You should test that the machines that host your source cluster brokers have connectivity to the Confluent Cloud cluster, as described in Test connectivity to Confluent Cloud. Under some circumstances, you can also test connectivity from Confluent Cloud VPC or VNet to your source cluster using a tool like the AWS VPC Reachability Analyzer.
- If the source cluster is in a cloud VPC or VNET and the Confluent Cloud cluster’s networking type is VPC Peered or VNET Peered, the Confluent Cloud VPC must be peered to the VPC that hosts the source cluster.
- If the source cluster is in a cloud VPC or VNET and the Confluent Cloud cluster’s networking type is VPC Peered or VNET Peered, the Transit Gateway routing must be configured to allow the Confluent Cloud VPC to reach the VPC that hosts the source cluster, and vice versa.
- If the source cluster is not hosted in a public cloud—for instance, if it is in an on-premises datacenter––make sure that you are using AWS Transit Gateway or GCP Route Import to provide connectivity between your cluster’s host machines and Confluent Cloud.
AUTHENTICATION_ERROR¶
Meaning: The source cluster security credential provided to the cluster link was not able to authenticate with the source cluster.
Solutions:
Confirm the security configuration that you assigned your cluster link.
For a Confluent Cloud source cluster, confirm that your link configuration has these properties:
If using API keys: -
security.protocol=SASL_SSL
-sasl.mechanism=PLAIN
-sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<source cluster API key>' password='<source cluster API secret>';
If using OAuth: - same parameters as your consumers use to authenticate with OAuth on the source cluster
For a Confluent Platform or Kafka source cluster, verify that the cluster link principal used in your link configuration is using an authentication mechanism that is enabled on the source cluster.
INVALID_BOOTSTRAP_INTERNAL_ENDPOINT_ERROR¶
Meaning: Your source cluster bootstrap server was to a private internal endpoint or port that Cluster Linking is not allowed to access in Confluent Cloud.
Solution: Verify that your bootstrap server uses port 9092
.
TIMEOUT_ERROR¶
Meaning: The operation to create the cluster link timed out. Solution: Contact Confluent Support.
UNKNOWN¶
Meaning: An unexpected error has occurred. Solution: Contact Confluent Support.
Troubleshoot and Task Error Codes¶
The following error codes are shared by cluster link and mirror topic commands to indicate problems with the link tasks or the mirror transition tasks.
With the exception of INTERNAL_ERROR
, these are all user-created errors.
In other words, if the error code is INTERNAL_ERROR
, then Confluent will have been alerted
and will be working on fixing the issue. Otherwise, the error is customer created and
the customer can and should alert on, as the error requires customer action to resolve.
Task Status | Description |
---|---|
UNKNOWN |
Error cause cannot be determined. |
INTERNAL_ERROR |
System error caused by Confluent software. This type of error automatically alerts Confluent, and resolution is in work. |
AUTHENTICATION_ERROR |
Authentication credentials are not properly configured. |
AUTHORIZATION_ERROR |
Authorization credentials are not properly configured. |
BROKER_AUTHENTICATION_ERROR |
Authentication credentials on the broker are not properly configured Confluent Platform. |
BROKER_AUTHORIZATION_ERROR |
Authorization credentials on the broker are not properly configured Confluent Platform. |
MISCONFIGURATION_ERROR |
A misconfiguration is causing errors. |
REMOTE_LINK_NOT_FOUND_ERROR |
The remote link was unexpectedly not found. |
LINK_NOT_FOUND_ERROR |
The cluster link cannot be found. |
CONSUMER_GROUP_IN_USE_ERROR |
The consumer group is active on the destination, causing offsets to not be synced. |
SECURITY_DISABLED_ERROR |
No authorizer is configured on the source cluster. |
TOPIC_EXISTS_ERROR |
A topic exists on the destination unexpectedly. |
POLICY_VIOLATION_ERROR |
The topic transition violates a policy. |
LINK_COORDINATOR_NOT_ENABLED_ERROR |
Cluster link is not enabled. |
ACL_LIMIT_EXCEEDED |
ACLs limit on the link has been exceeded. |
REMOTE_MIRROR_NOT_FOUND_ERROR |
Remote mirror topic is not available. |
UNKNOWN_TOPIC_OR_PARTITION_ERROR |
Either a topic or partition was unexpectedly not found. |
INVALID_TOPIC |
An InvalidTopicException was encountered from the destination cluster. This error would occur, for example, if the auto-create mirror task tries to create a topic on the destination cluster and the topic name is invalid. See the error message for more details. |
SUPPRESSED_ERRORS |
This means some errors were suppressed because too many were encountered. |
INVAILD_REQUEST_ERROR |
An InvalidRequestException was encountered. See the error message for more details. |