Use VPC Peering Connections with Confluent Cloud on AWS¶

A virtual private cloud (VPC) peering is a networking connection between your VPC and Confluent Cloud that enables you to route traffic using private IPv4 addresses. VPCs can communicate with each other as if they are within the same network.

For more information about VPC peering with AWS, see Introduction to Amazon VPC.

Managed connectors created in a VPC-peered cluster can access data sources and sinks hosted in all peered VPCs, if the firewall rules allow connector traffic to and from the peered VPCs.

When you set up a VPC peering connection between AWS VPC and Confluent Cloud using /27 CIDR blocks, the following Confluent Cloud feature is supported:

Fetch from Follower, a cost optimization to allow clients to consume from the nearest follower, instead of the leader

Requirements and considerations¶

A Confluent Cloud network of type PEERING in AWS.

If a network does not exist, see Confluent Cloud Network on AWS.
All AWS availability zones, except use1-az3 in the us-east-1 region, are supported.
Transitive VPC peering is not supported.

If you peer Network A to Network B, and peer Network B to Confluent Cloud, applications running in Network A will not be able to access Confluent Cloud. Although they don’t provide transitive routing, shared AWS VPCs can be leveraged to enable Confluent Cloud connectivity. For more information, see AWS Working with Shared VPCs.

To achieve transitivity, you can link an AWS Transit Gateway to a Confluent Cloud cluster in AWS.
You can have multiple VPC peering connections. For information about limits, see Network quotas in Confluent Cloud.
You can colocate multiple Confluent Cloud Dedicated clusters in the same Confluent Cloud network, but this is limited by the expected number and size of the clusters. The applicable limits are specified in Network.
Cross-region peering is not supported through the Confluent Cloud Console. Contact Confluent Support to see if your regions are supported and to request configuration.

Create a VPC peering connection¶

This section describes how to create a VPC peering connection to a Confluent Cloud network on AWS using the Confluent Cloud Console, Confluent REST API, or Confluent CLI.

See Terraform configuration for creating a AWS peering connection using Terraform.

The following information is required:

The AWS account ID associated with the VPC you are peering to Confluent Cloud network.
The VPC ID you are peering with Confluent Cloud network.
The CIDR block of the VPC you are peering with Confluent Cloud network. See CIDR blocks in Confluent Cloud network for the requirement details.
You might need to increase your route quota when you use VPC peering because the Confluent Cloud and AWS routes are shared.

Follow this procedure to create a VPC network peering connection to a Confluent Cloud cluster on AWS.

Create a peering connection from your VPC to the Confluent Cloud network:
1. In the Confluent Cloud Console, go to your Confluent Cloud network resource and click + VPC Peering.
2. Enter Name, AWS Account Number, AWS VPC ID, AWS VPC CIDR, and click Add.
  
  The AWS VPC CIDR value should not be identical and not completely within the Confluent Cloud network CIDRs. For example, with the Confluent Cloud network CIDR of 10.0.0.0/16, 10.0.0.0/8 is a valid AWS VPC CIDR, but 10.0.0.0/24 is not a valid AWS VPC CIDR.
  
  Peering connection provisioning will take a few minutes to complete. Your peering connection status will transition from “Provisioning” to “Inactive” in the Confluent Cloud Console.
A peering connection must be created from your VPC to the Confluent Cloud network in order to access Confluent Cloud clusters and services in the Confluent Cloud network.

REST request
POST https://api.confluent.cloud/networking/v1/peerings
REST authentication

See Authentication.

REST request body
{ "spec":{ "display_name":"My-Peering-1", "cloud":{ "kind":"AwsPeering", "account":"000000000000", "vpc":"vpc-00000000000000000", "routes":[ "10.0.0.0/16" ], "customer_region":"us-west-2" }, "environment":{ "id":"env-y0000w" }, "network":{ "id":"n-000000" } } }
The routes should not be identical and not completely within the Confluent Cloud network CIDRs. For example, with the Confluent Cloud network CIDR of 10.0.0.0/16, 10.0.0.0/8 is a valid route, but 10.0.0.0/24 is not a valid route.
Use the confluent network peering create Confluent CLI command to create a peering connection:
confluent network peering create aws-peering <flags>
The following command-specific flags are supported:
- --network: Required. Confluent Cloud network ID.
- --cloud: Required. The cloud provider. Set to aws.
- --cloud-account: Required. AWS account ID associated with the VPC that you are peering with Confluent Cloud network.
- --virtual-network: Required. AWS VPC ID that you are peering with Confluent Cloud network.
- --customer-region: Cloud region ID of the AWS VPC that you are peering with Confluent Cloud network.
- --aws-routes: Required. A comma-separated list of CIDR blocks of the AWS VPC that you are peering with Confluent Cloud network.
You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --context and --environment.

The following is an example Confluent CLI command to create a VPC peering:
confluent network peering create aws-peering \ --network n-123456 \ --cloud aws \ --cloud-account 123456789012 \ --virtual-network vpc-1234567890abcdef0 \ --aws-routes 172.31.0.0/16,10.108.16.0/21
When the connection status is “Inactive” in the Confluent Cloud Console, go to the Amazon VPC Console and accept the peering request. You have seven days to accept the request before it expires. For details on accepting peering connections, refer to Create and accept VPC peering connections in Amazon Virtual Private Cloud in the AWS documentation.

If your request has expired, contact Confluent to resend the request. After you have accepted the peering request, the status of the peering connection will change to “Active”.
Add the new peering connection to the route table for your VPC in the AWS Management Console. For details on updating route tables, see the AWS documentation.

For the routing to become effective, the route table must be associated with subnet(s).
1. Go to the VPC section of the AWS Management Console and click Route Tables.
2. Select the route table for your VPC and click Edit routes.
3. Click Add route.
4. Add the Confluent Cloud network VPC CIDR block to the Destination column. You can find the Confluent Cloud network VPC CIDR block in the Network Overview of your Confluent Cloud network.
5. Add the AWS Peering Connection ID to the Target column. This value is prefixed with pcx-.
6. Click Save routes.
When you are done, the VPC peering status should display “Active” in the Confluent Cloud Console.

VPC Peering using /27 CIDR blocks¶

Important

Limited Availability

Support for /27 CIDR blocks is in Limited Availability to a subset of Confluent customers. To be considered for access before General Availability, contact Confluent Support.

After creating a VPC Peering Confluent Cloud network that uses /27 CIDR blocks, perform the following steps to ensure that traffic can flow between your VPCs and the Confluent Cloud network:

Add routes for all three /27 CIDR blocks to all your VPC route tables, not just the zone-aligned routes.
Update the ACLs on your VPCs as needed.

Configure DNS forwarding¶

To resolve hostnames that reside within private DNS zones or a self-hosted DNS server and access your own VPC or on-prem from Confluent Cloud, set up DNS forwarding in Confluent Cloud.

For example, you can use DNS forwarding for Confluent Cloud fully-managed connectors that need to access data in your VPC.

DNS forwarding requires VPC peering or TGW connection where there is bi-directional network access between your network and Confluent Cloud clusters.

Step 1: Get DNS resolver IP addresses¶

To use the DNS forwarding feature with your AWS VPC, you can set up AWS Inbound Endpoints or use your own DNS server:

If you wish to forward DNS requests from Confluent Cloud to a Route53 hosted zone, create Inbound Endpoints for Confluent Cloud network to access your DNS servers.

AWS recommends deploying multiple endpoints in different availability zones for availability reasons.

For details, see Configuring inbound endpoints.

Once the endpoints are created, as described in the next step, in Confluent Cloud, input the IP addresses of the Inbound Endpoints to which to forward requests.
If you want to use your self-hosted DNS server, use the IP address of that DNS server in Confluent Cloud in the next step.

Step 2: Create a DNS Forwarder in Confluent Cloud¶

Set up DNS forwarding in Confluent Cloud:

In Confluent Cloud, navigate to the DNS Forwarding tab on the Network Detail page.
Input the following information:
- DNS server IPs: Up to 3 IP addresses of your DNS servers to which we should forward DNS requests.
- Domain list: Up to 10 domains to which you wish to route the DNS requests.
Wait until provisioning is complete and DNS is propagated.

Send a request to create a DNS Forwarder resource:

REST request

POST https://api.confluent.cloud/networking/v1/dns-forwarders

REST request body

{
  "spec":
  {
    "display_name": "<The Custom name for the DNS Resolver>",
    "environment":
    {
      "id": "<The Environment ID where the DNS Resolver belongs to>"
    },
    "config":
    {
      "kind": "ForwardViaIp",
      "dns_server_ips": "<A list of IP address(es), up to 3, of DNS server(s) from your VPC>"
    },
    "domains": "<A list of domains, up to 10, for the DNS forwarder to use>",
    "gateway":
    {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

To get the gateway id, issue the following API request:

GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}

You can find the gateway id in the response under spec.gateway.id.

Use the confluent network dns forwarder create Confluent CLI command to set up a DNS forwarder:

confluent network dns forwarder create <dns-forwarder-name> <flags>

The following command-specific flags are supported:

--dns-server-ip: Required. A comma-separated list of IP addresses for the DNS server.
--gateway: Required. Gateway ID. To get the gateway id, run the following CLI command:
```
confluent network describe
```
--domains: A comma-separated list of domains for the DNS forwarder to use.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --context, --environment, and --output.

The following is an example Confluent CLI command to create a DNS forwarder:

confluent network dns forwarder create \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

The following is an example Confluent CLI command to create a named DNS forwarder:

confluent network dns forwarder create  my-dns-forwarder \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

Next steps¶

Try Confluent Cloud on AWS Marketplace with $400 of free usage for 30 days, and pay as you go.