Use Google Cloud Private Service Connect

Google Cloud Private Service Connect allows for one-way secure connection access from your VPC to Confluent Cloud with an added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity of setup.

Benefits of using Private Service Connect with your Confluent Cloud clusters include:

  • A secure, unidirectional connection to Confluent Cloud that must be initiated from your VPC network.
  • Use the Google Cloud Console to configure DNS resolution for your private endpoints and allow traffic from your private endpoints to your Confluent Cloud service attachment.
  • Registered Google Cloud project IDs ensure auto-approval of connection requests when you use the published service attachment URI.
  • You don’t need to coordinate CIDR ranges between your network and Confluent Cloud.

The following diagram summarizes the Private Service Connect architecture that includes your VPC (or project) and the Confluent Cloud VPC (or account).

Private Service Connect architecture between customer VPC or project and Confluent Cloud cluster

Prerequisites

  • A Confluent Cloud network (CCN) of type PRIVATELINK in Google Cloud. If the network does not exist, follow the procedure below in Create a Confluent Cloud network in Google Cloud.
  • To use a Private Service Connect with a Confluent Cloud cluster, your VPC must allow outbound internet attachments for DNS resolution, Confluent Cloud Schema Registry, and Confluent CLI.
    • DNS requests to the public authority traversing to private hosted zone is required.
    • Confluent Cloud Schema Registry is accessible over the internet.
    • Provisioning new ksqlDB clusters requires Internet access. After ksqlDB clusters are up and running, they are fully accessible over Private Service Connect endpoints.
    • Confluent CLI requires internet access to authenticate with the Confluent Cloud control plane.
  • Confluent Cloud Console components, like topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with Private Service Connect, see Access Confluent Cloud Console with Private Networking.

Warning

For limitations of the Google Cloud Private Service Connect feature, see Limitations below.

Create a Confluent Cloud network in Google Cloud

To create a Dedicated cluster with Private Service Connect, you first need to create a Confluent Cloud network in the required cloud and region.

Note

You can create multiple clusters within one Confluent Cloud network. For details on default service quotas, see Available service quotas.

  1. In the Confluent Cloud Console, go to the Network management page for your environment.
  2. Click Create your first network if this is the first network in your environment, or click + Create network if your environment has existing networks.
  3. Select Google Cloud as the Cloud Provider and select a region from the Region list, and then click Next.
  4. Select the Private Service Connect connectivity type, select three zones for your network under Zone placement, and then click Continue. The Create Network page displays.
  5. In the Network name field, enter a meaningful label to help identify your network and review your selections under Network configuration. Optionally, click Review payment method to verify the payment method and promotions.
  6. Click Create Network.

Your network is created. If creating this network exceeds the network quota for your environment, you must delete an existing network or file a support ticket to increase your network quota.


Creating your Confluent Cloud network typically requires 15 to 20 minutes.

After successfully provisioning your Confluent Cloud network, you can add Dedicated clusters within your Confluent Cloud network by using either of the following procedures:

Register your Google Cloud project with Confluent Cloud

Register your Google Cloud projects with the Confluent Cloud network for automatic approval of private endpoint connections to the Confluent Cloud network. If required, you can register multiple projects.

  1. Open the Confluent Cloud Console, go to your network resource in the Network Management tab and click + Private Service Connect Access.
  2. Enter the Google Cloud project ID containing the VPCs that you want to connect from using Private Service Connect.
  3. Click Save.

Your Private Service Connect status transitions from “Pending” to “Active” in the Confluent Cloud Console. You still need to configure the private endpoints in your VPC before you can connect to the cluster.

Note the Private Service Connect endpoints to create an Private Service Connect connection from your VPC to the Confluent Cloud cluster. This URL is also provided later.

Connect to your Confluent Cloud cluster with Private Service Connect

Follow this procedure to create an Private Service Connect connection to a Confluent Cloud cluster on Google Cloud using the Confluent Cloud Console or REST APIs.

Note

Confluent recommends using a Terraform configuration for setting up Private Service Connect endpoints. This configuration automates the manual steps described below.

Create VPC endpoints for Private Service Connect

After the Private Service Connect access status is “Active” in the Confluent Cloud Console, you must configure private endpoints for your VPC by using the Google Cloud Console to connect from your VPC to Confluent Cloud using Private Service Connect.

  • For Confluent Cloud single zone clusters, you need to create a single Private Service Connect endpoint to the Confluent Cloud Service Attachment URI.
  • For Confluent Cloud multi-zone clusters, you need to create a Private Service Connect endpoint to each of the Confluent Cloud zonal Service Attachment URIs.
Prerequisites

In the Confluent Cloud Console, find the following information for your Confluent Cloud cluster under the Cluster Settings section and Confluent Cloud network under Confluent Cloud Network overview:

  • Kafka Bootstrap (in the General tab)
  • Zone IDs (in the Networking tab)
  • Service Attachment URI (in the Networking tab)
  • DNS Domain Name (in the Networking tab)
  • Zonal DNS Subdomain Names (in the Networking tab)
  1. Open Google Cloud Console and go the the VPC Dashboard at https://console.cloud.google.com/net-services/psc.

  2. In the CONNECTED ENDPOINTS tab, click + CONNECT ENDPOINT. The Create endpoint page appears.

  3. Under Target, select Published service.

  4. Enter the Service Attachment URI from your Network overview in the Confluent Cloud Console ast the Target Service.

  5. Enter an Endpoint name for your Private Service Connect endpoint.

  6. Select a Network to place this endpoint.

  7. Select a Subnetwork to place this endpoint. Because subnets are regional resources in Google Cloud, you should select a subnet in the same region as your Confluent Cloud network.

  8. Select or create an IP Address to assign this endpoint.

  9. Click Add endpoint.

    Wait for acceptance by Confluent Cloud. This happens almost immediately, in less than one minute. After the endpoint is accepted, it transitions from “Pending” to “Active”.

  10. Note the zones for your Confluent Cloud cluster from the Networking tab in the Confluent Cloud Console. Select the service in these zones. Ensure the desired subnet is selected for each zone. Failure to add all zones as displayed in the Confluent Cloud Console can cause connectivity issues to brokers in the omitted zones, which can result in an unusable cluster.

  11. Wait for acceptance by Confluent Cloud. This should happen almost immediately (less than a minute). Upon acceptance, the endpoint transitions from “Pending” to “Active”.

Set up DNS records to use Google Cloud VPC endpoints

You need to update your DNS configurations to ensure connectivity passes through Private Service Connect in the supported pattern. Any DNS provider that can ensure DNS is routed as shown can be used (Cloud DNS from Google is used in the example). For details on using Cloud DNS, see Set up DNS records for a domain name with Cloud DNS.

To update the DNS configuration using the Google Cloud Console:

  1. In the Google Cloud Console, go to Cloud DNS at https://console.cloud.google.com/net-services/dns/zones and create the Private Zone.

    1. Click + CREATE ZONE.
    2. For Zone Type, select Private.
    3. Enter a Zone name.
    4. For DNS name, enter the Confluent Cloud DNS. You can find this value in the Confluent Cloud Console.
    5. Click Create.
  2. Create the DNS records.

    1. Go to the Private Zone resource created above.

    2. Click + Add Record Set. The Create record set page appears.

    3. Create the required DNS record sets for your Confluent Cloud network.

      For Confluent Cloud multi-zone networks, create a bootstrap DNS record set and one DNS record set for each zone. The bootstrap record set is used for the initial Kafka bootstrap request and is not zonal (can be routed to any of the private endpoints).

      Note: The IP address for each Private Service Connect endpoint can be found under its associated network interface.

      Bootstrap record set

      DNS Name: *
      Resource Record Type: A
      TTL: 1
      TTL Unit: minutes
      IPv4 Address: <IPv4 address of all virtual endpoints created above>

      Zone record sets (one for each zone)

      DNS Name: *.<zone-ID>
      Resource Record Type: A
      TTL: 1
      TTL Unit: minutes
      IPv4 Address: <IPv4 address of virtual endpoint created above>
    4. Click CREATE.

    5. Verify the Private Zone is in use by the desired Network(s).

Validate connectivity to Confluent Cloud

  1. From an instance within the VPC (or anywhere the previous step’s DNS is set up), run the following to validate Kafka connectivity through Google Cloud Private Service Connect is working correctly.

    1. Set the BOOTSTRAP environment variable with the cluster bootstrap URL.

      export BOOTSTRAP=$<bootstrap-server-url>
      

      The bootstrap URL displayed in Confluent Cloud Console includes the port (9092). The value of the BOOTSTRAP environment variable should include the full hostname, but do not include the port. This lets you can run the openssl s_client -connect <host>:<port> command with the required values.

      For example:

      export BOOTSTRAP=lkc-xxxxxx-xxxxxx.asia-southeast1.gcp.glb.confluent.cloud
      
    2. Test connectivity to your cluster by running the openssl s_client -connect <host>:<port> command, specifying the $BOOTSTRAP environment variable for the <host> value and 9092 for the <port> value.

      openssl s_client -connect $BOOTSTRAP:9092 -servername $BOOTSTRAP -verify_hostname $BOOTSTRAP </dev/null 2>/dev/null | grep -E 'Verify return code|BEGIN CERTIFICATE' | xargs
      

      To run the openssl s_client -connect command, the -connect option requires that you specify the host and the port number. For details, see the openssl s_client documentation.

    3. If the return output includes -----BEGIN CERTIFICATE----- Verify return code: 0 (ok), connectivity to the bootstrap is confirmed.

      Note

      You might need to update the network security tools and firewalls to allow connectivity. If you have issues connecting after following these steps, confirm which network security systems your organization uses and whether their configurations need to be changed. If you still have issues, run the debug connectivity script and provide the output to Confluent Support for assistance with your Private Service Connect setup.

  2. Next, verify connectivity using the Confluent Cloud CLI.

    1. Sign in to Confluent CLI with your Confluent Cloud credentials.

      confluent login
      
    2. List the clusters in your organization.

      confluent kafka cluster list
      
    3. Select the cluster (with Private Service Connect) that you want to test.

      confluent kafka cluster use ...
      

      Example:

      confluent kafka cluster use lkc-a1b2c
      
    4. Create a cluster API key to authenticate with the cluster.

      confluent api-key create --resource ... --description ...
      

      Example:

      confluent api-key create --resource lkc-a1b2c --description "connectivity test"
      
    5. Select the API key you just created.

      confluent api-key use ... --resource ...
      

      Example:

      confluent api-key use WPCMCIQWLJCGYR5P --resource lkc-a1b2c
      
    6. Create a test topic.

      confluent kafka topic create test
      
    7. Start consuming events from the test topic.

      confluent kafka topic consume test
      
    8. Open another terminal tab or window.

    9. Start a producer.

      confluent kafka topic produce test
      
    10. Type anything into the produce tab and hit Enter; press Ctrl+D or Ctrl+C to stop the producer.

    11. The tab running consume prints what was typed in the tab running produce.

If the test is successful, your cluster is ready for use.

Limitations

Private Service Connect is supported for Dedicated clusters only.

Existing Confluent Cloud clusters cannot be converted to use Private Service Connect.

Connectors

Fully-managed Confluent Cloud connectors can connect to sources or sinks using a public IP address. Sources or sinks in the customer network with private IP addresses are not supported.

Single zone clusters

Each Confluent Cloud single zone cluster that uses a Private Service Connect service attachment is provisioned with service endpoints in one zone. The zone is selected based on Confluent Cloud placement policies.

To ensure connectivity over Private Service Connect, provision subnets in your VPC that minimally include the single zone in which the Private Service Connect service endpoint is provisioned.

Multi-zone clusters

Each Confluent Cloud multi-zone cluster that uses a Private Service Connect service attachment is provisioned with service endpoints in three zones. For Google Cloud regions that include more than three zones, zones are selected based on Confluent Cloud placement policies.

To ensure connectivity over Private Service Connect, provision subnets that minimally include the three zones in which the Private Service Connect endpoints are provisioned.

Google Cloud

Google Cloud does not support the following features for Private Service Connect:

  • Cross-region Private Service Connect: For a cluster in one region, you cannot use a Private Service Connect endpoint in a different region.
  • Cross-region Private Service Connect: If you have a Private Service Connect endpoint in one region, you cannot access that endpoint from a different region.
  • On-premises connectivity to private endpoints using Cloud Interconnect

See also: Prerequisites.