Use Google Cloud Private Service Connect with Confluent Cloud¶

Google Cloud Private Service Connect allows for one-way secure connection access from your VPC to Confluent Cloud clusters with an added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity of setup.

Benefits of using Private Service Connect with your Confluent Cloud clusters include:

A secure, unidirectional connection to Confluent Cloud that must be initiated from your VPC network.
Use the Google Cloud Console to configure DNS resolution for your private endpoints and allow traffic from your private endpoints to your Confluent Cloud service attachment.
Registered Google Cloud project IDs ensure auto-approval of connection requests when you use the published service attachment URI.
You don’t need to coordinate CIDR ranges between your network and Confluent Cloud.

The following diagram summarizes the Private Service Connect architecture that includes your VPC (or project) and the Confluent Cloud VPC (or project).

Private Service Connect architecture between customer VPC or project and Confluent Cloud cluster

To set up to use Google Cloud Private Service Connect with Confluent Cloud:

Register your Google Cloud project with Confluent Cloud.
Create a Private Service Connect connection to Confluent Cloud.
Set up DNS records to use Google Cloud VPC endpoints.
Validate connectivity to Confluent Cloud.

Requirements and considerations¶

Private Service Connect is only available for use with Dedicated clusters.
Existing Confluent Cloud clusters cannot be converted to use Private Service Connect.
Have a Confluent Cloud network (CCN) of type Private Service Connect in Google Cloud availabe. If the network does not exist, see Confluent Cloud Network on Google Cloud.
To use Google Cloud Private Service Connect with Confluent Cloud, your VPC must allow outbound internet connection for Confluent Cloud Schema Registry, ksqlDB, and Confluent CLI to work.
- Confluent Cloud Schema Registry is accessible over the internet.
- Provisioning new ksqlDB clusters requires Internet access. After ksqlDB clusters are up and running, they are fully accessible over Private Service Connect endpoints.
- Confluent CLI requires internet access to authenticate with the Confluent Cloud control plane.
Confluent Cloud Console components, such as topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with Private Service Connect, see Use Confluent Cloud with Private Networking.
Cross-region Google Cloud Private Service Connect connections are not supported.

Connectors¶

Fully-managed Confluent Cloud connectors can connect to sources or sinks using a public IP address. Sources or sinks in the customer network with private IP addresses are not supported.

Single zone clusters¶

Each Confluent Cloud single zone cluster that uses a Private Service Connect service attachment is provisioned with service endpoints in one zone. The zone is selected based on Confluent Cloud placement policies.

To ensure connectivity over Private Service Connect, provision subnets in your VPC that minimally include the single zone in which the Private Service Connect service endpoint is provisioned.

Multi-zone clusters¶

Each Confluent Cloud multi-zone cluster that uses a Private Service Connect service attachment is provisioned with service endpoints in three zones. For Google Cloud regions that include more than three zones, zones are selected based on Confluent Cloud placement policies.

To ensure connectivity over Private Service Connect, provision subnets that minimally include the three zones in which the Private Service Connect endpoints are provisioned.

Register your Google Cloud project with Confluent Cloud¶

Register your Google Cloud projects with the Confluent Cloud network for automatic approval of private endpoint connections to the Confluent Cloud network. If required, you can register multiple projects.

Open the Confluent Cloud Console, in the Network Management tab, click the Confluent Cloud network you want to add the connection to.
Click + Private Service Connect Access.
Enter the Google Cloud project ID containing the VPCs that you want to connect from using Private Service Connect.
Click Save.

HTTP POST request

POST https://api.confluent.cloud/networking/v1/private-link-accesses

Authentication

See Authentication.

Request specification

In the request specification, include Confluent Cloud network ID, project, environment, and, optionally, add the display name. You can use the following example, but update the attributes with your values.

{
   "spec": {
       "display_name": "psc-access",
       "cloud": {
           "kind": "GcpPrivateServiceConnectAccess",
           "project": "my-gc-project"
       },
       "environment":{
           "id":"env-o2568o"
       },
       "network": {
           "id":"n-gq11o6"
       }
   }
}

Your Private Service Connect status transitions from “Pending” to “Active” in the Confluent Cloud Console. You still need to configure the private endpoints in your VPC before you can connect to the cluster.

Note the Private Service Connect endpoints to create a Private Service Connect connection from your VPC to the Confluent Cloud cluster. This URL is also provided later.

Create a Private Service Connect connection to Confluent Cloud¶

Follow this procedure to create a Private Service Connect connection to a Confluent Cloud cluster on Google Cloud using the Confluent Cloud Console or REST API.

Set up the VPC endpoint for Private Service Connect in Google Cloud Console¶

After the Private Service Connect access status is “Active” in the Confluent Cloud Console, you must configure private endpoints for your VPC by using the Google Cloud Console to connect from your VPC to Confluent Cloud using Private Service Connect.

For Confluent Cloud single zone clusters, you need to create a single Private Service Connect endpoint to the Confluent Cloud Service Attachment URI.
For Confluent Cloud multi-zone clusters, you need to create a Private Service Connect endpoint to each of the Confluent Cloud zonal Service Attachment URIs.

Note

Confluent recommends using a Terraform configuration for setting up Private Service Connect endpoints. This configuration automates the manual steps described below.

Prerequisites¶

In the Confluent Cloud Console, find the following information for your Confluent Cloud cluster under the Cluster Settings section and Confluent Cloud network under Confluent Cloud Network overview:

Kafka Bootstrap (in the General tab)
Zone IDs (in the Networking tab)
Service Attachment URI (in the Networking tab)
DNS Domain Name (in the Networking tab)
Zonal DNS Subdomain Names (in the Networking tab)

Steps¶

Open Google Cloud Console and go the the VPC Dashboard at https://console.cloud.google.com/net-services/psc.
In the CONNECTED ENDPOINTS tab, click + CONNECT ENDPOINT. The Create endpoint page appears.
Under Target, select Published service.
Enter the Service Attachment URI from your Network overview in the Confluent Cloud Console ast the Target Service.
Enter an Endpoint name for your Private Service Connect endpoint.
Select a Network to place this endpoint.
Select a Subnetwork to place this endpoint. Because subnets are regional resources in Google Cloud, you should select a subnet in the same region as your Confluent Cloud network.
Select or create an IP Address to assign this endpoint.
Click Add endpoint.

Wait for acceptance by Confluent Cloud. This happens in less than a minute. After the endpoint is accepted, it transitions from “Pending” to “Active”.
Note the zones for your Confluent Cloud cluster from the Networking tab in the Confluent Cloud Console. Select the service in these zones. Ensure the desired subnet is selected for each zone. Failure to add all zones as displayed in the Confluent Cloud Console can cause connectivity issues to brokers in the omitted zones, which can result in an unusable cluster.
Wait for acceptance by Confluent Cloud. This should happen almost immediately (less than a minute). Upon acceptance, the endpoint transitions from “Pending” to “Active”.

Set up DNS records to use Google Cloud VPC endpoints¶

You need to update your DNS configurations to ensure connectivity passes through Private Service Connect in the supported pattern. Any DNS provider that can ensure DNS is routed as shown can be used (Cloud DNS from Google is used in the example). For details on using Cloud DNS, see Set up DNS records for a domain name with Cloud DNS.

DNS resolution options¶

For Google Cloud Private Service Connect Confluent Cloud networks, you can use the public or private DNS resolution:

The private DNS resolution is the recommended option and guarantees fully private DNS resolution.
The public DNS resolution is useful when you want to ensure that Confluent deployments are homogenous and conform to DNS configurations for your networks.

DNS resolution is selected when you create a Confluent Cloud network, and it cannot be modified after creating the Confluent Cloud network. See Create a Confluent Cloud network on Google Cloud.

Public DNS resolution¶

The public (also known as chased private in Confluent Cloud) DNS resolution is used for the bootstrap server and broker hostnames of a Confluent Cloud cluster that is using Google Cloud Private Service Connect. When the public resolution is used, the clusters in this network requires both public and private DNS to resolve cluster endpoints.

Only the the Confluent Global DNS Resolver (GLB) endpoints are advertised.

The public DNS resolution performs the following two-step process:

The Confluent Cloud Global DNS Resolver removes the glb subdomain and returns a CNAME for your bootstrap and broker hostnames.

Example: $lkc-id-$nid.$region.$cloud.glb.confluent.cloud

CNAME returned: $lkc-id.$nid.$region.$cloud.confluent.cloud
The CNAME resolves to your VPC private endpoints based on the Private DNS Zone configuration.

Private DNS resolution¶

When the Private DNS resolution option is enabled, your private DNS zone provides internal DNS resolution for your private networks without requiring external resolution to the Confluent Global DNS Resolver (GLB).

When the private DNS resolution is used, the clusters in this network only require private DNS to resolve cluster endpoints. Only non-GLB endpoints are advertised.

Tip

To identity the CNAME DNS zone records to correctly map to zonal endpoints for Confluent Cloud, you can run the DNS helper shell script.

Configure DNS zones¶

DNS entries need to be created for Private Service Connect irrespective of the DNS resolution option you selected when creating the Confluent Cloud network.

To update the DNS configuration using the Google Cloud Console:

In the Google Cloud Console, go to Cloud DNS at https://console.cloud.google.com/net-services/dns/zones and create the Private Zone.
1. Click + CREATE ZONE.
2. For Zone Type, select Private.
3. Enter a Zone name.
4. For DNS name, enter the Confluent Cloud DNS. You can find this value in the Confluent Cloud Console in the Confluent Cloud network detail page.
5. Click Create.
Create the DNS records.
1. Go to the Private Zone resource created above.
2. Click + Add Record Set. The Create record set page appears.
3. Create the required DNS record sets for your Confluent Cloud network.
  
  For Confluent Cloud multi-zone networks, create a bootstrap DNS record set and one DNS record set for each zone. The bootstrap record set is used for the initial Kafka bootstrap request and is not zonal (can be routed to any of the private endpoints).
  
  Note: The IP address for each Private Service Connect endpoint can be found under its associated network interface.
  
  Bootstrap record set
  
  DNS Name: *
  
  Resource Record Type: A
  
  TTL: 1
  
  TTL Unit: minutes
  
  IPv4 Address: <IPv4 address of all virtual endpoints created above>
  
  Zone record sets (one for each zone)
  
  DNS Name: *.<zone-ID>
  
  Resource Record Type: A
  
  TTL: 1
  
  TTL Unit: minutes
  
  IPv4 Address: <IPv4 address of virtual endpoint created above>
4. Click CREATE.
5. Verify the Private Zone is in use by the desired Network(s).

Validate connectivity to Confluent Cloud¶

From an instance within the VPC (or anywhere the previous step’s DNS is set up), run the following to validate Kafka connectivity through Google Cloud Private Service Connect is working correctly.
1. Set the BOOTSTRAP environment variable with the cluster bootstrap URL.
```
export BOOTSTRAP=$<bootstrap-server-url>
```
  The bootstrap URL displayed in Confluent Cloud Console includes the port (9092). The value of the BOOTSTRAP environment variable should include the full hostname, but do not include the port. This lets you can run the openssl s_client -connect <host>:<port> command with the required values.
  
  For example:
```
export BOOTSTRAP=lkc-xxxxxx-xxxxxx.asia-southeast1.gcp.glb.confluent.cloud
```
2. Test connectivity to your cluster by running the openssl s_client -connect <host>:<port> command, specifying the $BOOTSTRAP environment variable for the <host> value and 9092 for the <port> value.
```
openssl s_client -connect $BOOTSTRAP:9092 -servername $BOOTSTRAP -verify_hostname $BOOTSTRAP </dev/null 2>/dev/null | grep -E 'Verify return code|BEGIN CERTIFICATE' | xargs
```
  To run the openssl s_client -connect command, the -connect option requires that you specify the host and the port number. For details, see the openssl s_client documentation.
3. If the return output includes -----BEGIN CERTIFICATE----- Verify return code: 0 (ok), connectivity to the bootstrap is confirmed.
  
  Note
  
  You might need to update the network security tools and firewalls to allow connectivity. If you have issues connecting after following these steps, confirm which network security systems your organization uses and whether their configurations need to be changed. If you still have issues, run the debug connectivity script and provide the output to Confluent Support for assistance with your Private Service Connect setup.
Next, verify connectivity using the Confluent Cloud CLI.
1. Sign in to Confluent CLI with your Confluent Cloud credentials.
```
confluent login
```
2. List the clusters in your organization.
```
confluent kafka cluster list
```
3. Select the cluster (with Private Service Connect) that you want to test.
```
confluent kafka cluster use ...
```
  Example:
```
confluent kafka cluster use lkc-a1b2c
```
4. Create a cluster API key to authenticate with the cluster.
```
confluent api-key create --resource ... --description ...
```
  Example:
```
confluent api-key create --resource lkc-a1b2c --description "connectivity test"
```
5. Select the API key you just created.
```
confluent api-key use ... --resource ...
```
  Example:
```
confluent api-key use WPCMCIQWLJCGYR5P --resource lkc-a1b2c
```
6. Create a test topic.
```
confluent kafka topic create test
```
7. Start consuming events from the test topic.
```
confluent kafka topic consume test
```
8. Open another terminal tab or window.
9. Start a producer.
```
confluent kafka topic produce test
```
10. Type anything into the produce tab and hit Enter; press Ctrl+D or Ctrl+C to stop the producer.
11. The tab running consume prints what was typed in the tab running produce.

If the test is successful, your cluster is ready for use.