Manage Networking for Confluent Cloud Connectors¶
Consider the following when determining the public Internet access configuration for resources that fully-managed connectors must access. For Confluent Cloud networking details, see the Cloud Networking docs.
Networking¶
The following tabs provide network connectivity IP address details. Note that a Connect node runs in the same VPC/VNet as the cluster the Connect node was provisioned with. This is true for all cluster types (Basic, Standard, Enterprise and Dedicated). For Confluent Cloud networking details, see the Cloud Networking docs.
Public egress IP addresses are available on all the major cloud platforms. For details, see Public Egress IP Addresses for Confluent Cloud Connectors.
Public egress IP addresses are not supported with Custom Connectors.
The following information applies to a fully-managed Sink or Source connector connecting to an external system using a public IP address.
Cluster network type | Public IP address connectivity | IP range used by the connector |
---|---|---|
Public Endpoint | Yes | A set of public egress IP addresses (see Public Egress IP Addresses for Confluent Cloud Connectors) |
VPC Peering and Transit Gateway | Yes | Dynamic public IP/CIDR range from the cloud provider region where the Confluent Cloud cluster is located |
Private Link | Yes | Dynamic public IP/CIDR range from the cloud provider region where the Confluent Cloud cluster is located |
The following information applies to a fully-managed Sink or Source connector connecting to an external system using a private IP address.
Cluster network type | Private IP address connectivity | IP range used by the connector |
---|---|---|
VPC Peering and Transit Gateway | Yes. DNS Forwarding is required when a private DNS is in use. | The source IP address used is from the /16 CIDR range configured by the customer for the Confluent Cloud cluster |
Private Link | AWS: Yes using Egress Access Point [*] Azure, GCP: No |
The IP address of the load balancer which hosts the private link service |
Public Endpoint | No | N/A |
[*] The price premium will be $0.03/task/hour for Egress Access Points. For more information about Confluent Cloud connector pricing, see Confluent Pricinng.
See the following cloud provider documentation for additional information:
- Amazon Web Services IP address ranges
- Microsoft Azure IP address Ranges and Service Tags (PDF Download)
- Google Cloud IP address ranges
DNS considerations¶
The Domain Name System (DNS) is the system used to translate URLs/Hostnames to
IP addresses, for example, www.confluent.io
to 54.177.145.149
.
A public DNS server contains DNS records that can be resolved using the public internet. A private DNS server contains DNS records that can only be resolved in a private network, such as a VPC or an on-prem environment.
One way to check if a given hostname uses public DNS is running the dig
command with a public DNS resolver:
dig [DNS-server] <hostname>
DNS-server
can be any public DNS server, such as Google DNS server
(8.8.8.8
) and Cloudflare DNS server (1.1.1.1
).
For example:
dig 8.8.8.8 www.confluent.io
Fully managed connectors in Confluent Cloud support the following types of DNS zones/servers for resolving and accessing required endpoints.
AWS | Azure | Google Cloud | |
---|---|---|---|
Public DNS | Supported | Supported | Supported |
Private DNS | Supported with DNS Forwarding | Supported with DNS Forwarding | Not supported |
Service and gateway endpoints¶
Azure service endpoints and AWS gateway endpoints provide secure and direct private connectivity to Azure and AWS services over the cloud provider network backbone using an optimized route. These endpoints are located in the Confluent Cloud VPC/Vnet.
Managed connector network traffic is routed over cloud service provider (CSP) secure public endpoints for the following services:
- AWS
- Azure
- Azure Blob Storage
- Azure Cosmos DB
- Azure Event Hubs
- Azure Service Bus
- Microsoft SQL Server
Private hosted services and peered VPCs¶
Under certain scenarios you must use a proxy configuration to allow a connector to attach to a database, storage, or other service running on a private host. A specific scenario for using a proxy is when Kafka clusters are in a peered VPC configuration and a connector needs to attach to a non-peered VPC. This is because peering network connectivity is non-transitive so the connector cannot attach to endpoints in the non-peered VPC.
The following shows an example of this scenario on Google Cloud where a proxy configuration is required.
There are the following three VPCs running on Google Cloud:
- VPC A: The Confluent Cloud VPC.
- VPC B: The customer VPC.
- VPC C: The Cloud SQL VPC.
In this configuration, there is no transitive peering from VPC A to the private Cloud SQL database running on VPC C. For the connector to be able to attach to the database, a proxy client is added to VPC B so the connector can attach to a proxy server added to VPC C. For more information about setting up this Google Cloud proxy configuration, see How the Cloud SQL Auth proxy works.
Note
AWS Transit Gateway (TGW) clusters do not have this transitive issue unless there is a third VPC to connect to. For example, Confluent Cloud VPC A > TGW > Customer VPC B > Customer VPC C (or a cloud service VPC C as shown in the Google Cloud example scenario).
For Confluent Cloud networking details, see the Cloud Networking docs.
Troubleshoot networking issues for fully-managed connectors¶
This page describes common networking-related errors you may encounter when creating connectors, and it provides checklists that can help you to troubleshoot the issues.
Issues with Peering or Transit Gateway¶
Errors trying to connect via FQDN (fully qualified domain name) with publicly resolvable DNS
- If able to directly connect to the private IP address, there is an issue when resolving DNS.
- If not able to connect to the private IP address:
- Check the peering/Transit Gateway setup, routes, associated firewalls, security groups, and network access control lists.
- Check ports and protocol settings.
Errors trying to connect via FQDN with DNS that is not publicly resolvable
- Check if DNS forwarding is correctly set up with the right IP address for the DNS server and is forwarding the needed domain name. For details, see Configure DNS forwarding or Configure DNS forwarding.
- Check your DNS setup, peering/Transit Gateway setup, routes, associated firewalls, security groups, and network access control lists.
- Check ports and protocol settings.
Issues with Private Link¶
Errors related to a private endpoint when directly connecting to a private IP address
- Ensure that the Egress Access Point is correctly set up. For details, see AWS Egress Access Points for Dedicated Clusters.
- Check the associated firewalls, security groups, and network access control lists.
- Check ports and protocol settings.
Errors related to a private endpoint when directly connecting to an FQDN
- If the FQDN is publicly resolvable:
- Ensure that the Egress Access Point is correctly set up. For details, see AWS Egress Access Points for Dedicated Clusters.
- Check the associated firewalls, security groups, and network access control lists.
- Check ports and protocol settings.
- If the FQDN is not publicly resolvable:
- Ensure that the DNS record is set up for the Egress Access Point. For details, see Create a private DNS record in Confluent Cloud.
- Check the associated firewalls, security groups, and network access control lists.
- Check ports and protocol settings.