Use AWS Transit Gateway on Confluent Cloud¶

You can use AWS Transit Gateway to connect your VPCs to your Confluent Cloud clusters. The transit gateway acts as a cloud router, with each connection only made once. Your data is encrypted and never travels over the internet.

Requirements and considerations¶

To use AWS Transit Gateway with Confluent Cloud, you need the followings:

A Confluent Cloud network of type TRANSITGATEWAY on AWS.

After you create a Confluent Cloud network for your AWS Transit Gateway hub, you can:
- Add an AWS transit gateway attachment to a Confluent Cloud network
- Migrate from AWS VPC Peering to AWS Transit Gateway
  
  Existing Confluent Cloud clusters using secure public endpoints or AWS PrivateLink cannot be converted to use AWS Transit Gateway.
All AWS availability zones, except use1-az3 in the us-east-1 region, are supported.
Cluster Linking

Cluster links between two AWS Transit Gateway clusters are supported in the following CIDR blocks:
- RFC 1918
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16
- RFC 6598
  - 100.64.0.0/10

Add an AWS transit gateway connection¶

To create an AWS transit gateway:

In AWS, create a transit gateway.
In Confluent Cloud, add a transit gateway attachment.
In AWS, add a transit gateway attachment.
In AWS, update the route table.

Add an AWS transit gateway in AWS¶

In Confluent Cloud Console, in the Network management tab for your environment, click the Transit Gateway network resource to retrieve the Confluent Cloud AWS Account ID of the network resource you want to use.
In the Amazon VPC console, add an AWS Transit Gateway for your AWS account. For details, see Create a transit gateway.
In AWS Resource Access Manager (RAM) Console, create a resource share for your Confluent Cloud network.

For details, see Creating a resource share in AWS RAM and Share a transit gateway in the AWS documentation.

Use the Confluent Cloud AWS Account ID you retrieved in the first step as the principal of the resource share.
Save the Amazon Resource Name (ARN) of the resource share. You need to provide the name when you create a transit gateway attachment.

Add an AWS transit gateway attachment in Confluent Cloud¶

Add a transit gateway attachment in Confluent Cloud.

In the Confluent Cloud Console, select your environment, and click Network management.
In the Network management tab, select the AWS Transit Gateway network resource that you want to add a transit gateway attachment to, and click the Connections tab.
Click + Transit Gateway. The Add Transit Gateway Attachment page appears.
Specify the following information, and click Add.
- Name: Enter a name for your attachment.
- AWS RAM Share ARN: Use the ARN you saved in the last step of creating a resource share in AWS.
- AWS Transit Gateway ID: The identifier (ID) of the AWS Transit Gateway that you are connecting to the Confluent Cloud network.
- AWS VPC CIDR: A comma-separated list of destination CIDRs that you want to route to from the Confluent Cloud network when you attach Confluent Cloud to your Transit Gateway.
  
  This field needs to contain all CIDR blocks in your network (cloud VPC, on-prem) that needs return routes from the Confluent Cloud network via the Transit Gateway attachment.
  
  The RFC 1918 and RFC 6598 private address ranges are supported as described in Select CIDR blocks and block size.
  
  Public IP address ranges are not allowed.
  
  The routes should not be identical and not completely within the Confluent Cloud network CIDRs. For example, with the Confluent Cloud network CIDR of 10.0.0.0/16, 10.0.0.0/8 is a valid Transit Gateway route, but 10.0.0.0/24 is not a valid route.
Your Transit Gateway Attachment will transition to READY in the Confluent Cloud Console.

You may need to check if Auto accept shared attachments is configured on your Transit Gateway on AWS.

Here is an example REST request:

REST request

POST https://api.confluent.cloud/networking/v1/transit-gateway-attachments

REST authentication

See Authentication.

REST request body

In the request specification, include values for cloud, region, environment, connection type, and, optionally, add the display name, CIDR, and zones for the Confluent Cloud network. Update the attributes below with the correct values.

See CIDR blocks requirements for detail requirements.

{
   "spec":{
      "display_name":"prod-tgw-use1",
      "cloud":{
         "kind":"AwsTransitGatewayAttachment",
         "ram_share_arn":"arn:aws:ram:us-west-3:000000000000:resource-share/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
         "transit_gateway_id":"tgw-xxxxxxxxxxxxxxxxx",
         "routes":[
            "100.64.0.0/10",
            "10.0.0.0/8",
            "192.168.0.0/16",
            "172.16.0.0/12"
         ]
      },
      "environment":{
         "id":"env-abc123",
         "environment":"string"
      },
      "network":{
         "id":"n-00000",
         "environment":"string"
      }
   }
}

The routes are the IP address ranges routed back to your VPC Transit Gateway from Confluent Cloud when you attach Confluent Cloud to your Transit Gateway. This field needs to contain all CIDR blocks in your network (cloud VPC, on-prem) that need return routes from the Confluent Cloud network via the Transit Gateway attachment.

The RFC 1918 and RFC 6598 private address ranges are supported as described in Select CIDR blocks and block size, and public IP ranges are not allowed for routes.

The routes should not be identical and not completely within the Confluent Cloud network CIDRs. For example, with the Confluent Cloud network CIDR of 10.0.0.0/16, 10.0.0.0/8 is a valid Transit Gateway route, but 10.0.0.0/24 is not a valid Transit Gateway route.

Use the confluent network transit-gateway-attachment create Confluent CLI command to create an AWS transit gateway attachment:

confluent network transit-gateway-attachment create <attachment-name> <flags>

The following are the command-specific flags:

--network: Required. Confluent Cloud network ID.
--aws-ram-share-arn: Required. AWS Resource Name (ARN) for the AWS Resource Access Manager (RAM) Share of the AWS Transit Gateway that you want Confluent Cloud to be attached to.
--aws-transit-gateway: Required. The ID of the AWS Transit Gateway that you want Confluent Cloud to be attached to.
--routes: Required. A comma-separated list of destination CIDRs that you want to route to from the Confluent Cloud network when you attach Confluent Cloud to your Transit Gateway.

This needs to contain all CIDR blocks in your network (cloud VPC, on-prem) that need return routes from the Confluent Cloud network via the Transit Gateway attachment.

You can specify optional CLI flags described in the Confluent CLI command reference, such as --environment.

The following is an example Confluent CLI command to create a transit gateway attachment:

confluent network transit-gateway-attachment create my-tgw-attachment \
  --network n-123456 \
  --aws-ram-share-arn arn:aws:ram:us-west-2:123456789012:resource-share/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx \
  --aws-transit-gateway tgw-xxxxxxxxxxxxxxxxx \
  --routes 10.0.0.0/16,100.64.0.0/10

If your transit gateway attachment fails, delete the attachment and create a new one.

Attachment failures can occur for the following reasons:

The attachment is not shared with the correct AWS principal for the Confluent Cloud network.

Even if the auto-accept all attachment option is turned on, you still need to add the Confluent Cloud AWS Account ID to the AWS resource share you create in Add an AWS transit gateway in AWS.
The transit gateway was not added to the AWS resource share.
The AWS resource share ARN is not correct.

If you delete the transit gateway attachment, you must also delete the transit gateway route table entry for the Confluent Cloud network.

Add an AWS transit gateway attachment in AWS¶

To connect with the AWS transit gateway, add a transit gateway attachment in the AWS console as described in Create a transit gateway attachment to a VPC.

To connect with an on-premise networks, refer to Transit gateway attachments to a Direct Connect gateway.

Add the transit gateway route to the AWS route table¶

In the Amazon VPC console, add the Confluent Cloud CIDR to the route table with the target of your AWS transit gateway.

For details, see Update AWS route table.

Migrate from AWS VPC peering connections to AWS transit gateway attachments¶

Warning

When you migrate from VPC peering to a transit gateway, you cannot add the exact same CIDR route on the transit gateway as the CIDR that is used in your VPC peering connection.

To migrate from a Confluent Cloud network that uses AWS VPC Peering to a network using an AWS Transit Gateway hub, perform the following steps:

Establish a maintenance window.

Maintaining concurrent transit gateway attachments and VPC peering connections might result in a degraded experience due to asymmetric routing with MTU mismatches between the two connectivity types. Confluent recommends establishing a planned “maintenance window” during which both connection types exist to avoid packet losses for your critical workloads. While concurrent connections exist, traffic will be charged at AWS transit gateway rates.
File a Confluent Support ticket.

Because of the potential for degraded experience, migrations between VPC peering connections and transit gateway attachments on the same network require filing a Confluent Support ticket.

Sign in to the Confluent Support Portal at https://support.confluent.io/hc/, click SUBMIT A REQUEST, and then click AWS Transit Gateway Provisioning.
- Request to migrate your Confluent Cloud VPC Peering network (with existing VPC peering connections) to an AWS Transit Gateway network.
- Include your Confluent Cloud network name and network ID (available from the Confluent Cloud Console).
After Confluent Support temporarily enables concurrent connection types (for both VPC peering connections and transit gateway attachments), you will be informed that you can proceed with your migration, and you can continue to the next step.
For each existing VPC peering connection, perform the following steps:
1. In AWS, create a transit gateway.
2. In Confluent Cloud, add a transit gateway attachment.
3. In AWS, add a transit gateway attachment.
4. In the Amazon VPC console, update the AWS VPC route table.
  
  Edit the target of the associated routes to direct traffic to the transit gateway instead of the peering connection.
5. In the Confluent Cloud Console, delete the existing VPC peering connection.
6. In the AWS VPC console, verify that the AWS peering resource are deleted.
Warning

If your VPC resources are not deleted, asymmetric routing and packet losses occur, resulting in a degraded experience.
Validate connectivity over the newly provisioned resources.
After you complete the migration and validation steps, inform Confluent Support so that they can disable VPC peering connections and close the ticket.

Configure DNS forwarding¶

To resolve hostnames that reside within private DNS zones or a self-hosted DNS server and access your own VPC or on-prem from Confluent Cloud, set up DNS forwarding in Confluent Cloud.

For example, you can use DNS forwarding for Confluent Cloud fully-managed connectors that need to access data in your VPC.

DNS forwarding requires VPC peering or TGW connection where there is bi-directional network access between your network and Confluent Cloud clusters.

Step 1: Get DNS resolver IP addresses¶

To use the DNS forwarding feature with your AWS VPC, you can set up AWS Inbound Endpoints or use your own DNS server:

If you wish to forward DNS requests from Confluent Cloud to a Route53 hosted zone, create Inbound Endpoints for Confluent Cloud network to access your DNS servers.

AWS recommends deploying multiple endpoints in different availability zones for availability reasons.

For details, see Configuring inbound endpoints.

Once the endpoints are created, as described in the next step, in Confluent Cloud, input the IP addresses of the Inbound Endpoints to which to forward requests.
If you want to use your self-hosted DNS server, use the IP address of that DNS server in Confluent Cloud in the next step.

Step 2: Create a DNS Forwarder in Confluent Cloud¶

Set up DNS forwarding in Confluent Cloud:

In Confluent Cloud, navigate to the DNS Forwarding tab on the Network Detail page.
Input the following information:
- DNS server IPs: Up to 3 IP addresses of your DNS servers to which we should forward DNS requests.
- Domain list: Up to 10 domains to which you wish to route the DNS requests.
Wait until provisioning is complete and DNS is propagated.

Send a request to create a DNS Forwarder resource:

REST request

POST https://api.confluent.cloud/networking/v1/dns-forwarders

REST request body

{
  "spec":
  {
    "display_name": "<The Custom name for the DNS Resolver>",
    "environment":
    {
      "id": "<The Environment ID where the DNS Resolver belongs to>"
    },
    "config":
    {
      "kind": "ForwardViaIp",
      "dns_server_ips": "<A list of IP address(es), up to 3, of DNS server(s) from your VPC>"
    },
    "domains": "<A list of domains, up to 10, for the DNS forwarder to use>",
    "gateway":
    {
      "id": "<The gateway ID to which this belongs>",
      "environment": "<Environment of the referred resource, if env-scoped>"
    }
  }
}

To get the gateway id, issue the following API request:

GET https://api.confluent.cloud/networking/v1/networks/{Confluent Cloud network ID}

You can find the gateway id in the response under spec.gateway.id.

Use the confluent network dns forwarder create Confluent CLI command to set up a DNS forwarder:

confluent network dns forwarder create <dns-forwarder-name> <flags>

The following command-specific flags are supported:

--dns-server-ip: Required. A comma-separated list of IP addresses for the DNS server.
--gateway: Required. Gateway ID. To get the gateway id, run the following CLI command:
```
confluent network describe
```
--domains: A comma-separated list of domains for the DNS forwarder to use.

You can specify additional optional CLI flags described in the Confluent CLI command reference, such as --environment and --output.

The following is an example Confluent CLI command to create a DNS forwarder:

confluent network dns forwarder create \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

The following is an example Confluent CLI command to create a named DNS forwarder:

confluent network dns forwarder create  my-dns-forwarder \
  --domains abc.com,def.com \
  --dns-server-ips 10.200.0.0,10.201.0.0 \
  --gateway gw-123456

Next steps¶

Cluster Linking between AWS Transit Gateway attached Confluent Cloud clusters
Try Confluent Cloud on AWS Marketplace with $1000 of free usage for 30 days, and pay as you go. No credit card is required.