Use AWS Transit Gateway with Confluent Cloud¶

You can use AWS Transit Gateway to a single transit gateway to connect your VPCs to your Confluent Cloud clusters. The transit gateway acts as a cloud router, with each connection only made once. Your data is encrypted and never travels over the public internet.

Prerequisites¶

To use AWS Transit Gateway with Confluent Cloud, you need:

An AWS Transit Gateway is configured and ready to use for your AWS account. To create a transit gateway, see Create a transit gateway in the AWS documentation.
A resource share for your Confluent Cloud network must be created in your
AWS Resource Access Manager (RAM) Console. For details, see Creating a resource share in AWS RAM and Share a transit gateway in the AWS documentation.
- To share your AWS transit gateway with your Confluent Cloud network, use the Confluent Cloud AWS Account ID as the principal. In the Confluent Cloud Console, you can find the Confluent Cloud AWS Account listed in your Confluent Cloud network under Network overview.
Exact availability zone alignment between your AWS Transit Gateway, AWS transit gateway attachments, your VPC, and your Confluent Cloud network.
A Confluent Cloud network of type TRANSITGATEWAY in AWS. If you do not have a Confluent Cloud network, expand the following section and create one:
Create a Confluent Cloud network in AWS

To create a Confluent Cloud network for use with AWS Transit Gateway and transit gateway attachments, perform the following steps.

Note

You can create multiple clusters within one Confluent Cloud network. For details on default service quotas, see Network.

Before you create a Confluent Cloud network, you need the following information:
- Name for your Confluent Cloud network.
- Region and Availability Zones for the Confluent Cloud network. The Dedicated clusters created in these Confluent Cloud networks will inherit the Region and Availability Zones.
- CIDR block for the Confluent Cloud network. Review the following requirements for CIDR block selections.
  - The CIDR block must be in one of the following private networks, as mentioned in RFC 1918.
    - 10.0.0.0/8
    - 100.64.0.0/10
    - 172.16.0.0/12
    - 192.168.0.0/16
    - 198.18.0.0/15
  - The CIDR block cannot be any of the following:
    - 10.100.0.0/16
    - 10.255.0.0/16
    - 172.17.0.0/16
    - 172.20.0.0/16
  - Additional notes when selecting your CIDR block:
    - The RFC 6598 shared address space is supported on AWS.
    - Must be a /16 CIDR block.
    - Cannot be modified after the Confluent Cloud network is provisioned.
    - Must not overlap with an existing Confluent Cloud CIDR block.
1. In the Confluent Cloud Console, go to the Network management page for your environment.
2. Click Create your first network if this is the first network in your environment, or click + Add Network if your environment has existing networks.
3. Select AWS as the cloud service provider and select the geographic region in Region.
4. Select Transit Gateway as your connectivity type, enter your the Zone Placement and CIDR for Confluent Cloud, and then click Continue.
  
  Note that once your Confluent Cloud network is provisioned, you cannot change your selected Availability Zone (AZ) IDs. Make sure to deploy a network based on your zonal requirements.
5. Specify a Network Name, review your configuration, and click Create Network.
The following is an example REST API request:

HTTP POST request
POST https://api.confluent.cloud/networking/v1/networks
Authentication

See Authentication.

Request specification

Your REST request specification (spec) should include the following:
- display_name (optional) A meaningful name for your Confluent Cloud network.
- environment - id – The identifier (ID) of your Confluent Cloud environment.
- cloud – cloud service provider (AWS, Azure, or GCP)
- region – The Region where the network is located.
- connection_types – For transit gateway attachments, use TRANSITGATEWAY.
- zones – An array listing the three selected Availability Zone IDs in the same Region.
- cidr – The CIDR block.
Here is a REST specification example in JSON format. You can use this as a template for your own specification, replacing your unique values.
{ "spec": { "display_name": "aws-transit-gateway-network-us-west", "environment": { "id": "env-w85hlh" }, "cloud": "AWS", "region": "us-west-2", "connection_types": [ "TRANSITGATEWAY" ], "zones": [ "usw2-az1", "usw2-az2", "usw2-az3" ], "cidr": "100.64.0.0/16" } }
Your Confluent Cloud network is created and provisioned within 20 minutes. Take note of the Confluent Cloud network ID from the response to specify it in the following commands.

After successfully provisioning the Confluent Cloud network, you can add Dedicated clusters within your Confluent Cloud network by using either of the following procedures:
- Confluent Cloud Console: Create a Cluster in Confluent Cloud
- Cluster Management API: Create a cluster
Warning

For limitations on AWS Transit Gateway support, see Limitations below.

Add an AWS transit gateway attachment¶

After you create a Confluent Cloud network for your AWS Transit Gateway hub, you can:

Add an AWS transit gateway attachment to a Confluent Cloud network
Migrate from AWS VPC Peering to AWS Transit Gateway

Add an AWS transit gateway attachment to a Confluent Cloud network¶

Sign in to Confluent Cloud Console at https://confluent.cloud/login, go to your environment, and then click Network management.
Click the Confluent Cloud network (configured for AWS Transit Gateway) that you want to add a transit gateway attachment to, and then click the Connections tab.
Click + Transit Gateway. The Add Transit Gateway Attachment page appears.
On the Add Transit Gateway Attachment page, enter the following information, and then click Add.

Name: Enter a meaningful name for your attachment.

AWS RAM share ID: The Amazon Resource Name (ARN) of resource share for the transit gateway attachment. You can find the resource share ARN in your AWS Resource Access Manager console.

Example: arn:aws:ram:us-west-2:587051064079:resource-share/9d4ab048-9bde-4a9f-8cc6-d19a83635747

AWS Transit Gateway ID: The identifier (ID) of the AWS transit gateway that you are connecting to the Confluent Cloud network.

Enter the CIDR block for your AWS VPC in AWS VPC CIDR.

The CIDR block is the specific route used for the Confluent VPC and AWS Transit Gateway route tables.

Here is an example REST API request:

HTTP POST request

POST https://api.confluent.cloud/networking/v1/transit-gateway-attachments

Authentication

See Authentication.

Request specification

In the request specification, include values for cloud, region, environment, connection type, and, optionally, add the display name, CIDR, and zones for the Confluent Cloud network. Update the attributes below with the correct values.

{
   "spec":{
      "display_name":"prod-tgw-use1",
      "cloud":{
         "kind":"AwsTransitGatewayAttachment",
         "ram_share_arn":"arn:aws🐏us-west-3:000000000000:resource-share/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
         "transit_gateway_id":"tgw-xxxxxxxxxxxxxxxxx",
         "routes":[
            "100.64.0.0/10",
            "10.0.0.0/8",
            "192.168.0.0/16",
            "172.16.0.0/12"
         ]
      },
      "environment":{
         "id":"env-00000",
         "environment":"string"
      },
      "network":{
         "id":"n-00000",
         "environment":"string"
      }
   }
}

After provisioning, your new transit gateway attachment is available for use. Confirm your VPC and transit gateway routes, update your VPC routes, and validate that your transit gateway routes are correct.

Migrate from AWS VPC peering connections to AWS transit gateway attachments¶

Warning

Establish a maintenance window

Maintaining concurrent transit gateway attachments and VPC peering connections might result in a degraded experience due to asymmetric routing with MTU mismatches between the two connectivity types. Confluent recommends establishing a planned “maintenance window” during which both connection types exist to avoid packet losses for your critical workloads. While concurrent connections exist, traffic will be charged at AWS transit gateway rates.

Because of the potential for degraded experience, migrations between VPC peering connections and transit gateway attachments on the same network require filing a Confluent Support ticket.

To migrate from a Confluent Cloud network that uses AWS VPC Peering to a network using an AWS Transit Gateway hub, perform the following steps:

Sign in to the Confluent Support Portal at https://support.confluent.io/hc/, click SUBMIT A REQUEST, and then click AWS Transit Gateway Provisioning.
- Request to migrate your Confluent Cloud VPC Peering network (with existing VPC peering connections) to an AWS Transit Gateway network.
- Include your Confluent Cloud network name and network ID (available from the Confluent Cloud Console).
After Confluent Support temporarily enables concurrent connection types (for both VPC peering connections and transit gateway attachments), you will be informed that you can proceed with your migration, and you can continue to the next step.
For each existing VPC peering connection, perform the following steps:
1. In the Confluent Cloud Console, create an AWS transit gateway attachment.
2. In the Confluent Cloud Console, delete the existing VPC peering connection.
3. Go to your AWS VPC console at `https://console.aws.amazon.com/vpc/home and delete the AWS peering resource and the associated routes in your VPC route table.
Warning

If your VPC resources are not deleted, asymmetric routing and packet losses occur, resulting in a degraded experience.
Validate connectivity succeeds over the newly provisioned resources.
After you complete the migration and validation steps, inform Confluent Support so that they can disable VPC peering connections and close the ticket.

You have successfully migrated from using VPC peering connections to transit gateway attachments.

Validate connectivity to Confluent Cloud¶

From an instance within the VPC (or anywhere the previous step’s DNS is set up), run the following to validate Kafka connectivity through AWS PrivateLink is working correctly.
1. Set an environment variable with the cluster bootstrap URL.
```
export BOOTSTRAP=$<bootstrap-server-url>
```
  The Bootstrap URL displayed in Confluent Cloud Console includes the port (9092). The BOOTSTRAP value should include the full hostname, but do not include the port. This is so that you can run the openssl s_client -connect <host>:<port> command with the required values.
  
  For example:
```
export BOOTSTRAP=lkc-nkodz-0l6je.us-west-2.aws.confluent.cloud
```
2. Test connectivity to your cluster by running the openssl s_client -connect <host>:<port> command, specifying the $BOOTSTRAP environment variable for the <host> value and 9092 for the <port> value.
```
openssl s_client -connect $BOOTSTRAP:9092 -servername $BOOTSTRAP -verify_hostname $BOOTSTRAP </dev/null 2>/dev/null | grep -E 'Verify return code|BEGIN CERTIFICATE' | xargs
```
  To run the openssl s_client -connect command, the -connect option requires that you specify the host and the port number. For details, see the openssl s_client documentation.
3. If the return output is -----BEGIN CERTIFICATE----- Verify return code: 0 (ok), connectivity to the bootstrap is confirmed.
  
  Note
  
  You might need to update the network security tools and firewalls to allow connectivity. If you have issues connecting after following these steps, confirm which network security systems your organization uses and whether their configurations need to be changed. If you still have issues, run the debug connectivity script and provide the output to Confluent Support for assistance with your PrivateLink setup.
Next, verify connectivity using the Confluent Cloud CLI.
1. Sign in to Confluent CLI with your Confluent Cloud credentials.
```
confluent login
```
2. List the clusters in your organization.
```
confluent kafka cluster list
```
3. Select the cluster with AWS PrivateLink you wish to test.
```
confluent kafka cluster use ...
```
  For example:
```
confluent kafka cluster use lkc-a1b2c
```
4. Create a cluster API key to authenticate with the cluster.
```
confluent api-key create --resource ... --description ...
```
  For example:
```
confluent api-key create --resource lkc-a1b2c --description "connectivity test"
```
5. Select the API key you just created.
```
confluent api-key use ... --resource ...
```
  For example:
```
confluent api-key use WQDMCIQWLJDGYR5Q --resource lkc-a1b2c
```
6. Create a test topic.
```
confluent kafka topic create test
```
7. Start consuming events from the test topic.
```
confluent kafka topic consume test
```
8. Open another terminal tab or window.
9. Start a producer.
```
confluent kafka topic produce test
```
10. Type anything into the produce tab and hit Enter; press Ctrl+D or Ctrl+C to stop the producer.
11. The tab running consume will print what was typed in the tab running produce.

You’re done! The cluster is ready for use.

Limitations¶

Existing Confluent Cloud clusters cannot be converted to use AWS Transit Gateway.

Availability zones¶

All AWS availability zones, except use1-az3, are supported in the us-east-1 region.

Cluster linking¶

Cluster links between two Confluent Cloud Transit Gateway clusters require that both clusters are in one of the following CIDR blocks (RFC 1918):

10.0.0.0/8
100.64.0.0/10
172.16.0.0/12
192.168.0.0/16

The following CIDR blocks are incompatible with Transit Gateway-to-Transit Gateway cluster linking, even though they are valid Confluent Cloud CIDR blocks:

198.18.0.0/15

All CIDR blocks, however, are compatible with cluster linking between a public internet cluster and an AWS Transit Gateway cluster, or between a Confluent Platform (v. 7.0 or later) cluster and an AWS Transit Gateway cluster.

Suggested Resources¶

Cluster Linking between AWS Transit Gateway attached Confluent Cloud clusters