Use Google Cloud VPC Network Peering with Confluent Cloud¶

Google Cloud VPC Network Peering enables you to route traffic using private IPv4 addresses between your Virtual Private Cloud (VPC) and Confluent Cloud. VPC peers can communicate with each other as if they are within the same network.

For more information about, see VPC Network Peering.

Managed connectors created in a VPC-peered cluster can access data sources and sinks hosted in all peered VPCs, if the firewall rules allow connector traffic to and from the peered VPCs.

Requirements and considerations¶

A Confluent Cloud network of the “VPC Peering” type and the “Google Cloud” provider.

If a network does not exist, create a PEERING type network as described in Create Confluent Cloud Network on Google Cloud.
Your VPC subnets and Confluent Cloud must be in the same same region. Cross-region access to Confluent Cloud is not supported when VPC Network Peering is enabled with Google Cloud.
Transitive VPC peering is not supported. If you peer Network A to Network B, and peer Network B to Confluent Cloud, applications running in Network A will not be able to access Confluent Cloud.

Although they don’t provide transitive routing, shared Google Cloud VPCs can be leveraged to enable Confluent Cloud connectivity. For more information, see Google Cloud Shared VPC overview.
You can colocate multiple Confluent Cloud Dedicated clusters in the same Confluent Cloud network, but this is limited by the expected number and size of these clusters. The applicable limits are specified in Networks.
If using Google Cloud Organization policy constraints to restrict VPC Network Peering usage, contact Confluent Support to obtain the Confluent Cloud Organization ID in Google Cloud. Then whitelist the Organization ID in the constraints/compute.restrictVpcPeering policy in Google Cloud.

Create a VPC Network Peering connection¶

To establish a VPC Network Peering connection between Confluent Cloud and Google Cloud:

Create a VPC Network Peering connection in Confluent Cloud.
Create a VPC Network Peering connection in Google Cloud.

Step 1: Create a VPC Network Peering connection in Confluent Cloud¶

You need to gather the following information from the Google Cloud console:

The Google Cloud project ID associated with the VPC that you are peering to Confluent Cloud.
The network name of the VPC that you are peering with Confluent Cloud.
You might need to increase your route quota when you use VPC Network Peering because the Confluent Cloud and Google Cloud routes are shared.

Follow the steps to create a VPC Network Peering connection in Confluent Cloud.

In the Network Management tab of the desired Confluent Cloud environment, click the For dedicated cluster tab.
Click the Confluent Cloud network to which you want to add the peering connection.
In the Ingress connections tab, click + VPC Peering.
Specify the following field values.
- Name: : The name of this connection.
- GCP Project ID: The project ID associated with the VPC that you are peering to Confluent Cloud.
- GCP Network Name: The network name of the VPC that you are peering with Confluent Cloud.
- Import custom routes: Select to import static and dynamic custom routes over this VPC peering connection.
Click Add to create the peering connection.

Peering connection provisioning will take a few minutes to complete. Your peering connection status will transition from “Provisioning” to “Waiting for connection” in the Confluent Cloud Console.

A peering connection has to be created from your VPC to the Confluent Cloud network in order to access Confluent Cloud clusters and services in a Confluent Cloud network.

HTTP POST request

POST https://api.confluent.cloud/networking/v1/peerings

Authentication

See Authentication.

Request specification

{
   "spec":{
      "cloud":{
         "kind":"GcpPeering",
         "project":"my-project",
         "vpc_network":"gcp-vpc-peering",
         "import_custom_routes":false
      },
      "display_name":"My-GCP-Peering-1",
      "environment":{
         "id":"env-y0000w"
      },
      "network":{
         "id":"n-000000"
      }
   }
}

Use the confluent network peering create Confluent CLI command to create a peering connection:

confluent network peering create gcp-peering <flags>

The following command-specific flags are supported:

--network: Required. Confluent Cloud network ID.
--cloud: Required. The cloud provider. Set to gcp.
--cloud-account: Required. Google Cloud project ID associated with the VPC that you are peering with Confluent Cloud network.
--virtual-network: Required. Name of the Google Cloud VPC that you are peering with Confluent Cloud network.
--gcp-routes: Enable customer route import for Google Cloud VPC Network Peering.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create a VNet peering:

confluent network peering create gcp-peering \
  --network n-123456 \
  --cloud gcp \
  --cloud-account temp-123456 \
  --virtual-network customer-test-vpc-network \
  --gcp-routes

Use the confluent_peering resource to create a peering connection.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"

  lifecycle {
    prevent_destroy = true
  }
}

resource "confluent_network" "gcp-peering" {
  display_name = "GCP Peering Network"
  cloud = "GCP"
  region = "us-west4"
  cidr = "10.10.0.0/16"
  connection_types = ["PEERING"]
  environment {
    id = confluent_environment.development.id
  }

  lifecycle {
    prevent_destroy = true
  }
}

resource "confluent_peering" "gcp" {
  display_name = "GCP Peering"
  gcp {
    project = "temp-gear-123456"
    vpc_network = "customer-test-vpc-network"
  }
  environment {
    id = confluent_environment.development.id
  }
  network {
    id = confluent_network.gcp-peering.id
  }

  lifecycle {
    prevent_destroy = true
  }
}

See more Terraform configuration examples for creating a Google Cloud peering connection using Terraform:

Step 2: Create a VPC Network Peering connection to Confluent Cloud in Google Cloud¶

In order to complete setup, you must initiate a peering request to Confluent Cloud from Google Cloud VPC.

You need to gather the following information from the Confluent Cloud network and the peering connection you created in the previous section:

Confluent Cloud GCP Project ID: Your Confluent Cloud Project ID. You can find the value in the Confluent Cloud Console under Network overview of your Confluent Cloud network as the Confluent Cloud GCP Project ID.
VPC network name: Your Confluent Cloud VPC name. You can find the value in the Confluent Cloud Console under Network overview of your Confluent Cloud network as the Confluent Cloud VPC Network Name.

Follow the steps to create a VPC network peering connection in Google Cloud.

In VPC network in the Google Cloud Console, select VPC network peering.
Click CREATE CONNECTION to create a peering connection to Confluent Cloud.
Specify the following field values:
- Name: The name for this peering connection.
- Your VPC network: The name of your Google Cloud VPC network.
- Peered VPC network: Select In another project.
- Project ID: The Confluent Cloud GCP Project ID you retrieved in Confluent Cloud.
- VPC network name: The Confluent Cloud VPC name you retrieved in Confluent Cloud.
Click CREATE to initiate a peering connection to Confluent Cloud.
Verify that the new connection turns Active in Google Cloud.
Verify that the Status under VPC Peering connections is “Ready” in Confluent Cloud.

Import custom routes¶

The Import Custom Routes option in Confluent Cloud enables connectivity to a Confluent Cloud cluster from customer premise or other clouds, such as AWS and Azure. This connectivity is enabled by importing static and dynamic custom routes from a customer VPC into a Confluent Cloud VPC over the VPC Network Peering connection.

Review the considerations mentioned by Google Cloud in their VPC Network Peering documentation before enabling Import Custom Routes option.

The requirements and considerations for the Import Custom Routes option are:

The customer side VPC Network Peering has to be configured to export custom routes.
Transitive routing to your VPCs in the same or different regions is not supported.
- The only exception is when cross-regional VPCs are interconnected using Google Cloud VPN. However, your VPC, which is peered with Confluent Cloud cluster, must be in the same region as Confluent Cloud cluster.
- In Confluent Cloud, the global access option for the Google Cloud Internal LoadBalancer is not supported.
You cannot export Custom Routes from Confluent Cloud.
Privately used public IP (PUPI) addresses are not supported with the Import Custom Routes option.

The Import Custom Routes option must be enabled when you set up the VPC Network Peering connection.

To enable Import Custom Routes option on an existing VPC Network Peering connection:

Tear down the VPC peering connection and reestablish it with the Import Custom Routes option enabled.

Allow 15 minutes between tearing down the VPC connection and reestablishing it to avoid getting an error message during recreation.
To disable the Import Custom Routes option on an existing VPC Network Peering connection:

Tear down the VPC peering connection and reestablish it with the Import Custom Routes option disabled.

Allow 15 minutes between tearing down the VPC connection and reestablishing it to avoid getting an error message during recreation.

As an alternative, disable the Export Custom Route option in the customer VPC.

Configure DNS forwarding¶

To resolve hostnames that reside within private DNS zones or a self-hosted DNS server and access your own VPC or on-prem from Confluent Cloud, set up DNS forwarding in Confluent Cloud.

For example, you can use DNS forwarding for Confluent Cloud fully-managed connectors that need to access data in your VPC.

Confluent will forward DNS requests for a specified list of domains to Cloud DNS APIs of Google Cloud for resolution. You must grant Confluent’s IAM principal permission to do these lookups.

Step 1: Obtain Confluent’s IAM Principal¶

Confluent Cloud uses a unique IAM principal to make calls to the Cloud DNS API of Google Cloud for each environment you configure DNS forwarding for.

To obtain the IAM principal:

In Confluent Cloud, in the Network management tab, click For dedicated clusters.
Select the Confluent Cloud cluster you want to use.
Navigate to Cluster Overview -> Networking.
In the Network details section of the Peering type network you want to use, copy the value in the IAM principal field.

An example value: service-account@example.iam.gserviceaccount.com

Issue the following API request to get the gateway id.

HTTP GET request
```
GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}
```
Authentication

See Authentication.

You can find the gateway id in the response under spec.gateway.id.
Send a request to get the IAM principal id associated with the gateway. Use the gateway id you retrieved in the previous step.

HTTP GET request
```
POST https://api.confluent.cloud/networking/v1/gateways/{gateway-id}
```

Step 2: Grant Confluent’s IAM Principal read permissions in Cloud DNS¶

Next, you must grant Confluent’s IAM principal the DNS Reader role in each project that you have DNS queries to send to.

To grant permissions using the Google Cloud Console:

Navigate to the IAM & Admin product page.
Select the IAM sub-category.
In the Allow tab, click Grant Access.
In the New Principals field, paste the Confluent IAM principal that you retrieved in Step 1: Obtain Confluent’s IAM Principal.
In the Select a role field, select “DNS Reader”.
Click Save.

To grant permissions using the gcloud CLI, using the Confluent IAM principal that you retrieved in Step 1: Obtain Confluent’s IAM Principal:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
  --member "user: [Confluent IAM principal]" \
  --role="roles/dns.reader"

Step 3: Create a DNS Forwarder in Confluent Cloud¶

Set up DNS forwarding in Confluent Cloud.

Repeat the following steps for each domain you wish to forward.

In Confluent Cloud, navigate to the Confluent Cloud network and click For dedicated clusters.
In the DNS forwarding tab, click Add configuration.
Input the following information:
- Project ID: The project where the relevant private DNS zone is located.
- Zone name: The name of the DNS private zone.
- Domain: The domain for which you wish to forward requests.
Click Add and wait until provisioning is complete and DNS is propagated.

Get the gateway id using the following API request:
```
GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}
```
You can find the gateway id in the response under spec.gateway.id.

Send a request to create a DNS Forwarder resource:

REST request

POST https://api.confluent.cloud/networking/v1/dns-forwarders

REST request body

{
  "spec":
  {
    "display_name": "<The Custom name for the DNS Resolver>",
    "environment":
    {
      "id": "<The Environment ID where the DNS Resolver belongs to>"
    },
    "config":
    {
      "kind": "ForwardViaIp",
      "dns_server_ips": "<A list of IP address(es), up to 3, of DNS server(s) from your VPC>"
    },
    "domains": "<A list of domains, up to 10, for the DNS forwarder to use>",
    "gateway":
    {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

Use the confluent network dns forwarder create Confluent CLI command to set up a DNS forwarder:

confluent network dns forwarder create <dns-forwarder-name> <flags>

The following command-specific flags are supported:

--dns-server-ip: Required. A comma-separated list of IP addresses for the DNS server.
--gateway: Required. Gateway ID. To get the gateway id, run the following CLI command:
```
confluent network describe
```
--domains: A comma-separated list of domains for the DNS forwarder to use.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment and --output.

The following is an example Confluent CLI command to create a DNS forwarder:

confluent network dns forwarder create \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

The following is an example Confluent CLI command to create a named DNS forwarder:

confluent network dns forwarder create  my-dns-forwarder \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

Use the confluent_dns_forwarder Confluent Terraform Provider resource to set up a DNS forwarder.

An example snippet of Terraform configuration:

resource "confluent_environment" "development" {
  display_name = "Development"
}

resource "confluent_dns_forwarder" "main" {
  display_name = "dns_forwarder"
  environment {
    id = confluent_environment.development.id
  }
  domains = ["example.com", "domainname.com"]
  gateway {
    id = confluent_network.main.gateway[0].id
  }
  forward_via_ip {
    dns_server_ips = ["10.200.0.0", "10.200.0.1"]
  }
}

See Terraform configuration examples for a full example of setting up DNS forwarding using Terraform.

A DNS forwarder can be in one of the following states. A DNS forwarder is successfully created when it is in the CREATED or READY state.

PROVISIONING: DNS forwarder provisioning is in progress.
CREATED: The DNS forwarder is created. It will automatically become ready once a Kafka cluster is provisioned.
READY: The DNS forwarder is ready.
FAILED: The DNS forwarder is in a failed state.
DEGRADED: The DNS forwarder is in a degraded state, transitioning from READY due to unreachable DNS resolvers.
DEPROVISIONING: The DNS forwarder is being de-provisioned.