Use Google Cloud Private Service Connect with Confluent Cloud

Google Cloud Private Service Connect allows one-way secure connection access from your VPC to Confluent Cloud clusters with added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity of setup.

Benefits of using Private Service Connect with your Confluent Cloud clusters include:

• A secure, unidirectional connection to Confluent Cloud that must be initiated from your VPC network.
• Use the Google Cloud console to configure DNS resolution for your private endpoints and allow traffic from your private endpoints to your Confluent Cloud service attachment.
• Registered Google Cloud project IDs ensure auto-approval of connection requests when you use the published service attachment URI.
• You don’t need to coordinate CIDR ranges between your network and Confluent Cloud.

The following diagram summarizes the Private Service Connect architecture that includes your VPC (or project) and the Confluent Cloud VPC (or project).

To set up to use Google Cloud Private Service Connect with Confluent Cloud:

1. Identify a Confluent Cloud network you want to use, or set up a new Confluent Cloud network.
2. Add a Private Service Connect access in Confluent Cloud.
3. Provision Private Service Connect endpoints in Google Cloud.
4. Set up DNS records in Google Cloud.
5. Validate connectivity.
6. Troubleshoot broker connectivity issues if necessary.

The following tutorial video walks you through the process of configuring a private link. It also covers common mistakes and gotchas when setting up a private link with Confluent Cloud. Even though it uses AWS PrivateLink as an example, you will still find it relevant and useful for setting up Private Service Connect in Google Cloud.

Requirements and considerations

Review the following requirements and considerations before you set up a Private Service Connect in Google Cloud with Confluent Cloud:

• Private Service Connect is only available for use with Dedicated clusters.

• Have a Confluent Cloud network of type Private Service Connect in Google Cloud available. If the network does not exist, see Confluent Cloud Network on Google Cloud.

• To use Google Cloud Private Service Connect with Confluent Cloud, your VPC must allow outbound Internet connection for Confluent Cloud Schema Registry, ksqlDB, and Confluent CLI to work.

  • Confluent Cloud Schema Registry is accessible over the Internet.
  • Provisioning new ksqlDB clusters requires Internet access. After ksqlDB clusters are up and running, they are fully accessible over Private Service Connect endpoints.
  • Confluent CLI requires Internet access to authenticate with the Confluent Cloud control plane.

• Confluent Cloud Console components, such as topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with Private Service Connect, see Use Confluent Cloud with Private Networking.

• Cross-region attachments for Google Cloud Private Service Connect endpoints are not supported. Attachments must be made in the same region where Confluent Cloud is provisioned.

  Once attached, endpoints can be accessed cross-region by enabling Private Service Connect Global Access for the endpoint. For example, to enable global access for an endpoint in us-west1:

  gcloud compute forwarding-rules update cc-psc-us-west1-endpoint1 \
    --allow-psc-global-access --region=us-west1
  

  For details, see gcloud forwarding rules.

• Existing Confluent Cloud clusters cannot be converted to use Private Service Connect.

Connectors

Fully-managed Confluent Cloud connectors can connect to sources or sinks using a public IP address. Sources or sinks in the customer network with private IP addresses are not supported.

Single-zone clusters

Each Confluent Cloud single-zone cluster that uses a Private Service Connect service attachment is provisioned with service endpoints in one zone. The zone is selected based on Confluent Cloud placement policies.

To ensure connectivity over Private Service Connect, provision subnets in your VPC that minimally include the single-zone in which the Private Service Connect service endpoint is provisioned.

Multi-zone clusters

Each Confluent Cloud multi-zone cluster that uses a Private Service Connect service attachment is provisioned with service endpoints in three zones. For Google Cloud regions that include more than three zones, zones are selected based on Confluent Cloud placement policies.

To ensure connectivity over Private Service Connect, provision subnets that minimally include the three zones in which the Private Service Connect endpoints are provisioned.

Add a Private Service Connect access in Confluent Cloud

To make a connection to a cluster in Confluent Cloud using Private Service Connect, first, register your Google Cloud projects with the Confluent Cloud network for automatic approval of private endpoint connections to the Confluent Cloud network. If required, you can register multiple projects. This adds a Private Link access in Confluent Cloud. If required, you can register multiple subscriptions.

Confluent Cloud ConsoleConfluent REST APIConfluent CLI

1. In the Confluent Cloud Console, in the Network Management tab, click the Confluent Cloud network you want to add the connection to.
2. Click + Private Service Connect Access.
   • Name: The name for this access.
   • GCP Project ID: The Google Cloud project ID for the account containing the VPC that you want to connect from using Private Service Connect.
3. Click Add.

When your Private Service Connect status transitions to Ready in the Confluent Cloud Console, configure the private endpoints in your VPC before you can connect to the cluster.

Note the Private Service Connect endpoints to create a Private Service Connect connection from your VPC to the Confluent Cloud cluster. This URL is also provided later.

Provision Private Service Connect endpoints in Google Cloud

When the Private Service Connect access status becomes Ready in the Confluent Cloud Console, configure and attach Private Service Connect endpoints for your VPC by using the Google Cloud console.

• For Confluent Cloud single-zone clusters, you need to create a single Private Service Connect endpoint to the Confluent Cloud Service Attachment URI.
• For Confluent Cloud multi-zone clusters, you need to create three Private Service Connect endpoints, one endpoint to each of the Confluent Cloud zonal Service Attachment URIs.

Note

Confluent recommends using a Terraform configuration for setting up Private Service Connect endpoints. This configuration automates the manual steps described below.

To set up Private Service Connect endpoints:

1. In the Confluent Cloud Console in Cluster Overview, gather the following information:

   • In Cluster Settings:

     • Bootstrap server endpoint in the Endpoints section

     • Zones in the Cloud details section

       The availability zones of the Kafka cluster.

   • In Networking > Details:

     • DNS domain
     • VPC Endpoint service name
     • Service Attachment URI
     • Availability zone IDs
     • DNS subdomain

2. In the Google Cloud console, go to the VPC Dashboard at https://console.cloud.google.com/net-services/psc.

3. In the CONNECTED ENDPOINTS tab, click + CONNECT ENDPOINT.

   For details of the process, see Access published services through endpoints.

4. Specify the following values in the fields:

   • Target: Select Published service.

   • Target service: Enter the Service Attachment URI from your Network overview in the Confluent Cloud Console.

     For a Confluent Cloud single availability zone cluster, use the Service Attachment URI for the Kafka cluster availability zone.

     
     
     
     
     
     

     For a Confluent Cloud multi-availability zone cluster, use the Service Attachment URI for each connect endpoint. Note that you need to create three connect endpoints for the three availability zones.

   • Endpoint name: Give the name for this Private Service Connect endpoint.

   • Network: Specify the network to place this endpoint.

   • Subnetwork: Because subnets are regional resources in Google Cloud, you must select a subnet in the same region and availability zone as your Confluent Cloud network from the Networking tab in the Confluent Cloud Console.

     The zones for the Confluent Cloud network and cluster must match the zones of the VPC you want to make the Google Cloud Private service Connect connections from. You must have the matching subnets in your VPC for these zones so that IP addresses can be allocated from them.

     Failure to add all zones as displayed in the Confluent Cloud Console can cause connectivity issues to brokers in the omitted zones, which can result in an unusable cluster.

   • IP address: Specify the IP address to assign this endpoint.

5. Click ADD ENDPOINT.

   Wait for acceptance by Confluent Cloud. This happens in less than a minute.

Set up DNS records in Google Cloud

You need to update your DNS configurations to ensure connectivity passes through Private Service Connect in the supported pattern. Any DNS provider that can ensure DNS is routed as shown can be used (Cloud DNS from Google is used in the example). For details on using Cloud DNS, see Set up DNS records for a domain name with Cloud DNS.

DNS resolution options

For Google Cloud Private Service Connect Confluent Cloud networks, you can use the public or private DNS resolution:

• The private DNS resolution is the recommended option and guarantees fully private DNS resolution.
• The public DNS resolution is useful when you want to ensure that Confluent deployments are homogenous and conform to DNS configurations for your networks.

DNS resolution is selected when you create a Confluent Cloud network, and it cannot be modified after creating the Confluent Cloud network. See Create a Confluent Cloud network on Google Cloud.

Public DNS resolution

The public (also known as chased private in Confluent Cloud) DNS resolution is used for the bootstrap server and broker hostnames of a Confluent Cloud cluster that is using Google Cloud Private Service Connect. When the public resolution is used, the clusters in this network require both public and private DNS to resolve cluster endpoints.

Only the Confluent Global DNS Resolver (GLB) endpoints are advertised.

The public DNS resolution performs the following two-step process:

1. The Confluent Cloud Global DNS Resolver removes the glb subdomain and returns a CNAME for your bootstrap and broker hostnames.

   Example: $lkc-id-$nid.$region.$cloud.glb.confluent.cloud

   CNAME returned: $lkc-id.$nid.$region.$cloud.confluent.cloud

2. The CNAME resolves to your VPC private endpoints based on the Private DNS Zone configuration.

Private DNS resolution

When the Private DNS resolution option is enabled, your private DNS zone provides internal DNS resolution for your private networks without requiring external resolution to the Confluent Global DNS Resolver (GLB).

When the private DNS resolution is used, the clusters in this network only require private DNS to resolve cluster endpoints. Only non-GLB endpoints are advertised.

Create a DNS zone and DNS records

DNS entries need to be created for Private Service Connect irrespective of the DNS resolution option you selected when creating the Confluent Cloud network.

To create the DNS zone and DNS records for the Private Service Connect:

1. In the Google Cloud console, browse to the Cloud DNS page at https://console.cloud.google.com/net-services/dns/zones.

2. Create a DNS zone.

   1. Click + CREATE ZONE.
      • Zone Type: Select Private.
      • Zone name: Specify a name for this zone.
      • DNS name: Specify the Confluent Cloud DNS Domain. You can find this value in the Confluent Cloud Console on the Confluent Cloud network detail page.
   2. Click CREATE.

3. (Optional) To figure out the correct DNS zone records for specific VPC endpoints for Confluent Cloud, you can run the DNS helper shell script from your VM instance within the VPC.

   You specify the endpoint DNS domain name, Google Cloud project id, and one or more Confluent Cloud service attatchment URIs as arguments. You can find these values form the Networking detail page in the Confluent Cloud Console.

   The output contains the domain names, the record types, and the DNS values that you can input when you create DNS records. For example:

   ./dns-endpoints.sh lkc-1n0nvv-6k0qeg.us-central1.gcp.glb.confluent.cloud:9092 <my gcp project> <service attachment URI 1> <service attachment URI 2> <service attachment URI 3>
   
   Hosted zone domain: 6k0qeg.us-central1.gcp.confluent.cloud
   
   *                          A    10.0.0.2 10.0.0.3 10.0.0.4
   *.us-central1-a            A    10.0.0.2
   *.us-central1-b            A    10.0.0.3
   *.us-central1-c            A    10.0.0.4
   

4. Create DNS records.

   1. Go to the DNS zone you created in the previous step.

   2. In the RECORD SETS section, click + ADD STANDARD. The Create record set page appears.

   3. Create the required DNS record sets for your Confluent Cloud network.

      Create a bootstrap DNS record set and one DNS record set for each zone. The bootstrap record set is used for the initial Kafka bootstrap request and is not zonal (it can be routed to any of the private endpoints).

      The bootstrap DNS record should contain all three endpoints IPs. The IP address for each Private Service Connect endpoint can be found under its associated network interface.

      
      
      

      1. Create the bootstrap DNS record set.

         The record resolves too all three zonal endpoints that you created in Provision Private Service Connect endpoints in Google Cloud.

         • DNS name: *

         • Resource record type: A

         • TTL: 1

         • TTL unit: minutes

         • IPv4 Address: The IP addresses you specified for the connect endpoint in Provision Private Service Connect endpoints in Google Cloud.

           The bootstrap DNS record contains all three zonal endpoint IPs.

         
         

         The following is an example bootstrap DNS record.

         

      2. Create three zonal record sets, one for each availability zone of the Confluent Cloud network.

         • DNS name: *.<zone-ID>

           You get the <zone-ID> in Confluent Cloud under the Cluster settings page.

           

         • Resource record type: A

         • TTL: 1

         • TTL unit: minutes

         • IPv4 Address: The IP address you specified for the connect endpoint in Provision Private Service Connect endpoints in Google Cloud.

           The zonal records contain only the IP for the specific Private Service Connect endpoint.

         

      Note

      In Confluent Cloud with private linking, Kafka broker names you retrieve from the metadata are not static. Do not hardcode the broker names in DNS records.

      The following are example DNS record sets.

      

   4. Click CREATE.

   5. Verify the Private Zone is in use by the desired Network(s).

Your cluster is now ready for use. If you encounter any problem, refer to troubleshoot connectivity issues.