Use Google Cloud Egress Private Service Connect Endpoints for Dedicated Clusters on Confluent Cloud¶

Google Cloud Private Service Connect is a networking service that allows one-way connectivity from one VPC to a service provider and is popular for its unique combination of security and simplicity.

Confluent Cloud supports outbound Google Cloud Private Service Connect connections using Egress Private Service Connect Endpoints. Egress Private Service Connect Endpoints enable you to access supported Google Cloud services and other endpoint services powered by Google Cloud Private Service Connect from Confluent Cloud clusters. Examples of such services are Google Cloud Storage, a SaaS service, or a Private Service Connect Service that you create yourself.

The following diagram summarizes the Egress Private Service Connect Endpoint architecture between Confluent Cloud and various potential destinations.

Google Cloud Egress Private Service Connect Endpoint architecture

To set up an Egress Private Service Connect Endpoint from Confluent Cloud to an external system, such as for managed connectors:

Obtain the Google Cloud Private Service Connect Endpoint target.

For certain target systems, you can retrieve the service attachment URI as part of the guided workflow while creating an Egress Private Service Connect Endpoint in the next step.
Create an Egress Private Service Connect Endpoint in Confluent Cloud.
[Optional] Create private DNS records for use with Google Cloud Private Service Connect endpoints.

For service/connector-specific setup, see the target system networking supportability table.

Requirements and considerations¶

Review the following requirements and considerations before you set up an Egress Private Service Connect endpoint using Google Cloud Private Service Connect:

Egress Private Service Connect Endpoints described in this document are only available for use with Dedicated clusters.
The Google Cloud Private Service Connect service must be configured to allow access from Confluent Cloud’s Project ID.

Due to the differing granularity of the allowlist configuration across SaaS providers, it is recommended that you leverage provider-specific controls (like network rules) for securing access to the Private Service Connect services against confused deputy-type issues.
Egress Private Service Connect Endpoints can only be used by fully managed connectors.
Google Cloud does not support cross-region connections with Private Service Connect.
When using Egress Private Service Connect Endpoints, additional charges may apply, for example, for certain connector configurations. For more information, see the following pricing information:
- Confluent pricing
- Fully-managed Kafka Connector pricing

Obtain Google Cloud Private Service Connect Endpoint target¶

To make a Google Cloud Private Service Connect connection from Confluent Cloud to an external system, you must first obtain a Google Cloud Private Service Connect Endpoint target for Confluent Cloud to establish a connection to.

Depending on the system you wish to connect to, there may be different allowlist requirements to allow Confluent access. It is recommended that you check each system’s allowlist mechanism to verify that Confluent Cloud will be able to create an endpoint targeting that system.

For Google Cloud published services¶

Refer to the Google Cloud documentation for a list of all Google Cloud Google published services that integrate with Google Cloud Private Service Connect.

Confluent supports creating a Private Service Connect endpoint to the All APIs bundle for global Google APIs. In order to connect to this bundle, use the endpoint target value of all-google-apis.

For 3rd party services¶

Refer to the Google Cloud documentation for a list of Third-party published services that integrate with Google Cloud Private Service Connect.

And then refer to the system provider’s documentation about how to obtain the Private Service Connect service attachment URI to be used as the endpoint target and how to determine allowlisting requirements.

For Google Cloud Private Service Connect services you create¶

Refer to the Google Cloud documentation for how to make your endpoint service available to service consumers.

Manage access to your service¶

When you stand up your own Private Service Connect service, you may want to manage its permissions to restrict who can create endpoints to that service.

Confluent Cloud uses a unique Google Cloud Project to create Private Service Connect endpoints for each environment you create an Egress Private Service Connect Endpoint from. We highly recommend only accepting connections from this specific project for maintaining an optimal security posture.

You can obtain the Project ID during the endpoint creation process.

Create an Egress Private Service Connect endpoint in Confluent Cloud¶

Confluent Cloud Egress Private Service Connect Endpoints are used to connect to Google Cloud Private Service Connect services from Confluent Cloud.

In the Network Management tab of the desired Confluent Cloud environment, click the Confluent Cloud network to which you want to add the Private Service Connect Endpoint. The Connection Type of the network you select must be “Private Service Connect Access”.
Click + Create endpoint in the Egress connections tab.
Click the service you want to connect to:
- Global Google APIs
  
  Most Google APIs and services are available as part of Google’s All APIs bundle. Confluent will create a Private Service Connect Endpoint to the All APIs bundle when you configure using an endpoint target of ‘all-google-apis’.
- Snowflake, Elastic, MongoDB, or Other
  
  Select to connect to a different published service other than Google’s API bundle.
Follow the guided steps to specify the field values, including:
- Service: The name of the Private Service Connect service you retrieved in Obtain Google Cloud Private Service Connect Endpoint target.
  
  Need to specify if Other was selected in the previous step as the service type.
- Endpoint name: Name of the Private Service Connect Endpoint.
- Private Service Connect Endpoint Target:
  
  Need to specify if Other was selected in the previous step as the service type.
  
  If Global Google APIs was selected in the previous step, the value is preset to all-google-apis.
Click Create endpoint to create the Private Service Connect Endpoint.
Add a DNS record.

You have an option to create a DNS record in a separate workflow as described in Create a private DNS record in Confluent Cloud.
1. Specify the required value.
  - Domain: The domain of the Private Service Connect service you wish to access. Get the domain value from the Private Service Connect service provider, Google Cloud or a third-party provider.
2. Click Create DNS record.
Click Finish to complete the endpoint creation.

Send a request to create an endpoint:

HTTP POST request

POST https://api.confluent.cloud/networking/v1/access-points

Authentication

See Authentication.

Request specification

{
  "spec": {
    "display_name": "<The custom name for the endpoint>",
    "config": {
      "kind": "GcpEgressPrivateServiceConnectEndpoint",
      "private_service_connect_endpoint_target": "<The URI of the service attachment for the published service you want to connect to>",
      "target_system": "<The target system or service that the Private Service Connect Endpoint connects to>",
    },
    "environment": {
      "id": "<The environment ID where the endpoint belongs to>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    },
    "gateway": {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

vpc_endpoint_service_name: See Obtain Google Cloud Private Service Connect Endpoint target.
private_service_connect_endpoint_target: The URI of the service attachment for the published service that the Private Service Connect Endpoint connects to or "ALL_GOOGLE_APIS" for global Google APIs.
target_system: The target system or service that the Private Service Connect Endpoint connects to. For example, "GCS" or "SNOWFLAKE".
gateway.id: Issue the following API request to get the gateway id.
```
GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}
```
You can find the gateway id in the response under spec.gateway.id.

Use the confluent network access-point private-link egress-endpoint create Confluent CLI command to create an Egress Private Service Connect Endpoint:

confluent network access-point private-link egress-endpoint create [name] [flags]

The following are the command-specific flags:

--cloud: Required. The cloud provider. Set to gcp.
--service: Required. The URI of a GCP Private Service Connect target that you retrieved in Obtain Google Cloud Private Service Connect Endpoint target.

Or specify all-google-apis for the endpoints that connect to Global Google APIs.
--gateway: Required. Gateway ID.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

Use the confluent_access_point resource to create an Egress Private Service Connect Endpoint.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_access_point" "main" {
  display_name = "access_point"
  environment {
    id = confluent_environment.development.id
  }
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  gcp_egress_private_link_endpoint {
    private_service_connect_endpoint_target = "ALL_GOOGLE_APIS"
  }
}

Your Egress Private Service Connect Endpoint status will transition from “Provisioning” to “Ready” in the Confluent Cloud Console when the endpoint has been created and can be used.

Once the endpoint is created, connectors provisioned against Kafka clusters in the same network can leverage the Egress Private Service Connect Endpoint to access the external data.

Confluent Cloud exposes the Private Service Connect Endpoint ID for each of the above Egress Private Service Connect Endpoints so that you can use it in various network-related policies, such as in a Snowflake Network rule.

Create a private DNS record in Confluent Cloud¶

Not all service providers set up public DNS records to be used when connecting to them with Google Cloud Private Service Connect. For situations where a system provider requires setting up private DNS records in conjunction with Google Cloud Private Service Connect, you have to create DNS records in Confluent Cloud.

Before you create a DNS record, you need to first create an Egress Private Service Connect Endpoint and use the Egress Private Service Connect Endpoint ID for the DNS record.

In the Network Management tab of your environment, click the Confluent Cloud network you want to add the DNS record to.
In the Egress DNS tab, click Create DNS record.
Specify the following field values:
- Egress endpoint: The Egress Endpoint ID you created in the Create an Egress Private Service Connect Endpoint step.
- Domain: The domain of the Private Service Connect target you wish to access. Get the domain value from the Private Service Connect service provider, Google Cloud or a third-party provider.
Click Save.

Send a request to create a DNS Record that is associated with a Private Service Connect Endpoint that is associated with a gateway.

HTTP POST request

POST https://api.confluent.cloud/networking/v1/dns-records

Authentication

See Authentication.

Request specification

{
  "spec": {
    "display_name": "The name of this DNS record",
    "domain": "<The fully qualified domain name of the external system>",
    "config": {
      "kind": "PrivateLinkAccessPoint",
      "resource_id": "<The ID of the endpoint that you created>"
    },
    "environment": {
      "id": "<The environment ID where this resource belongs to>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    },
    "gateway": {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

domain: Get the value from the Private Service Connect service provider, Google Cloud, or a third-party provider.
gateway.id: Issue the following API request to get the gateway ID.
```
GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}
```
You can find the gateway ID in the response under spec.gateway.id.

An example request spec to create a DNS record:

{
  "spec": {
    "display_name": "prod-dns-record1",
    "domain": "example.com",
    "config": {
      "kind": "PrivateLinkAccessPoint",
      "resource_id": "plap-12345"
    },
    "environment": {
      "id": "env-00000",
    },
    "gateway": {
      "id": "gw-00000",
    }
  }
}

Use the confluent network dns record create Confluent CLI command to create a DNS record:

confluent network dns record create [name] [flags]

The following are the command-specific flags:

--private-link-access-point: Required. Private Service Connect Endpoint ID.
--gateway: Required. Gateway ID.
--domain: Required. Fully qualified domain name of the external system. Get the domain value from the Private Service Connect service provider, Google Cloud or a third-party provider.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

Use the confluent_dns_record Resource resource to create DNS records.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_dns_record" "main" {
  display_name = "dns_record"
  environment {
    id = confluent_environment.development.id
  }
  domain = "example.com"
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  private_link_access_point {
    id = confluent_access_point.main.id
  }
}

Support for Google Cloud Private Service Connect configuration¶

Confluent Support can help with issues you may encounter when creating an Egress Private Service Connect Endpoint to a specific service.

For any service-side problems, such as described below, Confluent is not responsible for proper Google Cloud Private Service Connect configuration or setup:

If you need help setting up a Google Cloud Private Service Connect for data systems running within your environment or VPC that you want to connect to from Confluent Cloud, contact Google Cloud for configuration help and best practices.
If you need help configuring Google Cloud Private Service Connect for those managed by a third-party provider or service, contact that provider for compatibility and proper setup.