Troubleshoot Cluster Linking on Confluent Platform

This page describes common errors you may encounter when creating cluster links, and how to address them. Task error codes for cluster link and mirror topic commands are also listed.

Cluster Linking error codes

Cluster links can return error codes during link creation or while an existing link is running. These error codes appear in creation response messages or when you describe a cluster link or mirror topic. Each error includes a customized message with specific context about the clusters involved.

Use the following guidelines to troubleshoot your issue:

UNRESOLVABLE_BOOTSTRAP_ERROR

Meaning

The destination cluster or, for a source-initiated link, the source cluster, could not resolve the provided bootstrap server to an IP address. In other words, this bootstrap server had no DNS entry in the cluster’s DNS.

Solutions

If the provided source cluster bootstrap server is a URL with a fully qualified domain name (FQDN) that is not Confluent Cloud (for example, kafka.example.com:9092), then the DNS is not properly configured for the destination cluster to resolve that FQDN to an IP address.

  • If the destination cluster is a Confluent Cloud cluster, this error is likely because your FQDN relies on your private DNS. You can’t provide Confluent Cloud with access to your private DNS. In this case, you have two options:

    • The best way to resolve this error is to use an FQDN that is registered in a public DNS. Alternatively, use an IP address (such as 10.10.4.24:9092) for the bootstrap server, instead of a host name. Your source cluster must have an IP address advertised listener on every broker before you can use Cluster Linking with an IP address.

    • Alternatively, you can register this FQDN with a public DNS, pointing to the appropriate IP address of your broker, proxy, or load balancer, so that Confluent Cloud can resolve this FQDN to an IP that it can reach. You can register internal IPs in a public DNS for the purposes of a load balancer.

  • If the destination cluster is a Confluent Platform cluster, then you can attempt to fix the destination cluster’s DNS. Ensure the route to your source cluster allows traffic over port 9092.

  • If the provided source cluster bootstrap server is for Confluent Cloud, for example, pkc-lmn0p.us-west-2.aws.confluent.cloud:9092, then contact Confluent Support.

BOOTSTRAP_TCP_CONNECTION_FAILED_ERROR

Meaning

The destination cluster cannot reach an Apache Kafka® cluster at the provided source bootstrap server.

Solutions

What if both clusters are Confluent Cloud clusters?

  • First verify that your source cluster is up and running, and in a healthy state. If it isn’t, this causes the error.

  • If the source cluster has internet networking, contact Confluent Support.

  • If the source cluster has Transit Gateway networking, follow the steps for troubleshoot Transit Gateway connectivity.

  • If the source cluster does not have internet networking or Transit Gateway networking, make sure that the two Confluent Cloud clusters exist in the same Confluent Cloud network, region, and cloud provider. To learn more about supported cluster combinations on Confluent Cloud, see Supported cluster types in the overview and Supported cluster combinations for private networking.

What if the source cluster is Confluent Platform or Kafka and the destination cluster is Confluent Cloud?

Providing network access to make Confluent Cloud clusters accessible is your responsibility. If your clusters cannot connect, you must resolve this.

  • For destination-initiated links, the Confluent Cloud cluster must be able to reach the source cluster.

  • For source-initiated links where the destination is Confluent Cloud, the on-premises source cluster must be able to reach the destination Confluent Cloud cluster.

If you intend to access a source cluster over the internet, validate that all your source cluster’s brokers have public internet IP addresses. You can verify this by consuming from your cluster, for example, with kafka-console-consumer, on a machine that does not have VPN or other private access to your cluster. For the consumer configuration, enter the bootstrap servers and security configuration that you provided to the cluster link.

Depending on your network configuration, validate the following:

  • Supported networking type: The Confluent Cloud cluster must use a supported private networking type to reach your private network. This includes VPC or VNet peering, AWS Transit Gateway or GCP Route Import, and PrivateLink or Private Service Connect (PSC), available for Enterprise and Dedicated clusters.

  • Connectivity testing: Test that the machines that host your source cluster brokers have connectivity to the Confluent Cloud cluster, as described in Test connectivity to Confluent Cloud. In some cases, you can also test connectivity from a Confluent Cloud VPC or VNet to your source cluster by using the AWS VPC Reachability Analyzer.

  • PrivateLink/Private Service Connect (PSC): Validate that outbound private endpoints (egress) are properly provisioned and mapped so Confluent Cloud can securely route traffic to your self-managed Confluent Platform or Kafka cluster.

  • VPC/VNet peering: If the source cluster is in a cloud VPC or VNet and the Confluent Cloud cluster uses VPC peering or VNet peering, the Confluent Cloud VPC must be peered to the VPC that hosts the source cluster.

  • Transit Gateway: If the source cluster is in a cloud VPC or VNet and the Confluent Cloud cluster uses Transit Gateway networking, configure routing to enable two-way communication between the Confluent Cloud VPC and the VPC hosting the source cluster.

  • On-premises environments: If the source cluster is not hosted in a public cloud, such as in an on-premises datacenter, use AWS Transit Gateway or GCP Route Import to provide connectivity between your cluster host machines and Confluent Cloud.

  • Firewall and ports: Verify that your security groups, firewalls, or ACLs allow inbound traffic from Confluent Cloud on the broker port, typically 9092 or 9094.

AUTHENTICATION_ERROR

Meaning

The security credentials provided to the cluster link could not authenticate with the source cluster.

Solutions

  • Confirm the security configuration that you assigned your cluster link.

  • For a Confluent Cloud source cluster, confirm that your link configuration has these properties:

    If using API keys: - security.protocol=SASL_SSL - sasl.mechanism=PLAIN - sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<source cluster API key>' password='<source cluster API secret>';

    If using OAuth: - same parameters as your consumers use to authenticate with OAuth on the source cluster

  • For a Confluent Platform or Kafka source cluster, verify that the cluster link principal (user or service account) used in your link configuration is using an authentication mechanism that is enabled on the source cluster.

INVALID_BOOTSTRAP_INTERNAL_ENDPOINT_ERROR

Meaning

Your source cluster bootstrap server points to a private internal endpoint or port that Cluster Linking cannot access in Confluent Cloud.

Solutions

Verify that your bootstrap server uses port 9092.

TIMEOUT_ERROR

Meaning

The operation to create the cluster link timed out.

Solutions

Contact Confluent Support.

UNKNOWN

Meaning

An unexpected error has occurred.

Solutions

Contact Confluent Support.

Task error codes

The following error codes apply to cluster link and mirror topic commands and signal problems with link tasks or mirror transition tasks.

With the exception of INTERNAL_ERROR, these are all user-created errors. In other words, if the error code is INTERNAL_ERROR, Confluent receives an alert and works to address the issue. Otherwise, the error is user-created and requires customer action to resolve.

Task Status

Description

UNKNOWN

Error cause cannot be determined.

INTERNAL_ERROR

System error caused by Confluent software. This type of error automatically alerts Confluent, and resolution is in work.

AUTHENTICATION_ERROR

Authentication credentials are not properly configured.

AUTHORIZATION_ERROR

Authorization credentials are not properly configured.

BROKER_AUTHENTICATION_ERROR

Authentication credentials on the broker are not properly configured Confluent Platform.

BROKER_AUTHORIZATION_ERROR

Authorization credentials on the broker are not properly configured Confluent Platform.

MISCONFIGURATION_ERROR

A misconfiguration is causing errors.

REMOTE_LINK_NOT_FOUND_ERROR

The remote link was unexpectedly not found.

LINK_NOT_FOUND_ERROR

The cluster link cannot be found.

CONSUMER_GROUP_IN_USE_ERROR

The consumer group is active on the destination, causing offsets to not be synced.

SECURITY_DISABLED_ERROR

No authorizer is configured on the source cluster.

TOPIC_EXISTS_ERROR

A topic exists on the destination unexpectedly.

POLICY_VIOLATION_ERROR

The topic transition violates a policy.

LINK_COORDINATOR_NOT_ENABLED_ERROR

Cluster link is not enabled.

ACL_LIMIT_EXCEEDED

ACLs limit on the link has been exceeded.

REMOTE_MIRROR_NOT_FOUND_ERROR

Remote mirror topic is not available.

UNKNOWN_TOPIC_OR_PARTITION_ERROR

Either a topic or partition was unexpectedly not found.

INVALID_TOPIC

An InvalidTopicException was encountered from the destination cluster. This error would occur, for example, if the auto-create mirror task tries to create a topic on the destination cluster and the topic name is invalid. See the error message for more details.

SUPPRESSED_ERRORS

This means some errors were suppressed because too many were encountered.

INVALID_REQUEST_ERROR

An InvalidRequestException was encountered. See the error message for more details.

Troubleshoot mirror topics

On Confluent Cloud and Confluent Platform, mirror command describe on a failed mirror topic returns the cause of the failure. In the case of a failed mirror topic, you have the following choices to remediate:

  • Failover or delete the mirror topic.

  • Contact Confluent Support to repair the failed mirror topic for a subset of the failures. Failure causes that can be repaired include the following:

    • UNSUPPORTED_MESSAGE_FORMAT: Records with message format v0 or v1 cannot be replicated using Cluster Linking because log divergence can’t be reliably detected with these older formats.

    • RECORD_TOO_LARGE: The request included a message larger than the maximum message size the server accepts.

    • TRUNCATION_BELOW_HIGH_WATERMARK: Truncation occurred below the high watermark. This can be caused by an unclean source leader election or an inability to detect source topic recreation.

Mirror topics can be transitioned in various ways, such as with promote, failover, or reverse. Both Confluent Cloud and Confluent Platform provide metrics and APIs that can help you find solutions when things go wrong. To learn more, see the following topics: