Egress PrivateLink Endpoints Setup Guide: RDS and Self-Managed Services on AWS for Confluent Cloud

This topic presents the steps for setting up Egress PrivateLink Endpoints for Amazon Relational Database Service (RDS) and self-managed services on AWS and Confluent Cloud to enable fully managed connectors in Confluent Cloud using AWS PrivateLink.

Note

The steps for RDS in this documentation are meant for testing purposes. For production, the recommendation is to utilize RDS Proxy as described in this document since RDS supports dynamic IPs and multi-zone type deployments.

Step 1. Create a gateway in Confluent Cloud (for Enterprise cluster only)

If using an Enterprise cluster in Confluent Cloud, you must set up a gateway in Confluent Cloud as described in Create a gateway for outbound connectivity in Confluent Cloud.

Step 2. Obtain instance details

• Obtain the instance details for RDS.

  1. Go to the instance page and note down the endpoint, the port, and the zone.

     The following are example entries for Postgres RDS.

     

  2. Obtain the private IP address using the following commands from the machine with access to the RDS instance.

     • In Windows: nslookup [rds-endpoint]
     • In Linux and MacOS: dig [rds-endpoint]

• Obtain the instance details for self-managed services.

  1. Go to the instance page and note down the private IP address and the zone of the instance.

     The following are example entries for an EC2 instance.

     

Step 3. Create a target group

1. In the AWS EC2 Dashboard, browse to Load Balancing → Target Groups.

2. Click Create target group.

3. Specify the group details:

   • Choose a target type: IP addresses
   • Target group name: The name of the target group
   • Protocol: TCP
   • Port: The port number used by the service (for example, 5432 for Postgres)
   • VPC: VPC where service is hosted
   • Health check protocol: TCP

4. Click Next.

5. Register the targets.

   1. Network: Select the correct network.
   2. Enter an IPv4 address from a VPC subnet: The private IP address of the instance captured above.
   3. Ports: check that the port number is correct.

6. Click Include as pending below.

7. Review targets.

   Ensure that proper private IP address, the correct port number, and the zone are listed for your service.

8. Click Create target group.

Step 4. Create a load balancer

1. In the AWS EC2 Dashboard, browse to Load Balancing → Load Balancers.
2. Click Create load balancer.
3. Click Create to create a Network Load Balancer.
4. Specify the Basic configuration settings.
   1. Load balancer name: The name of your load load balancer.
   2. Scheme: Internal
   3. IP address type: IPv4
5. Specify the Network mapping settings.
   1. VPC: Select the VPC where you want to deploy the load balancer.
   2. Mappings: Select the associated subnets.
   3. IPv4 address: Leave the Private IPv4 address field unchanged.
6. In the Security groups section, ensure that the inbound rule for the port number is applied.
7. Specify the Listeners and routing settings.
   1. Protocol: TCP
   2. Port: The port number of the service
   3. Forward to: The target created in the previous Step 3 section.
8. Click Create load balancer.
9. Wait for the load balancer status to change to “Active”.

Step 5. Create an endpoint service

1. In the AWS VPC Dashboard, browse to Virtual private cloud → Endpoint services.

2. Click Create endpoint service.

3. Specify the Endpoint service settings.

   1. Name: name endpoint service
   2. Load balancer type: Network

4. In the Specify Available load balancers section, select the network load balancer created in Step 4.

   

5. Specify the Additional settings.

   1. Require acceptance for endpoint: Acceptance required
   2. Supported IP address types: IPv4

6. Click Create.

7. Note the Service name of the created endpoint service.

   

Step 6. Allow Confluent principal

1. Obtain Confluent’s ARN in the Confluent Cloud Console.

   1. Browse to your environment and its network, and select the associated PrivateLink network.

   2. Click the Egress PrivateLink Endpoints tab.

      

2. In the AWS console, in the Allow principals tab of the created endpoint, click Allow principals.

3. Specify Confluent’s ARN.

   

Step 7. Create an Egress PrivateLink Endpoint

1. In the Network Management tab of the desired Confluent Cloud environment, click the Confluent Cloud network you want to add the PrivateLink Endpoint to. The Connection Type of the network needs to be “PrivateLink Access”.

2. Click Create endpoint in the Egress connections tab.

3. Click the service you want to connect to. Select Other if you do not see the specific service.

4. Follow the guided steps to specify the field values, including:

   • Name: Name of the PrivateLink Endpoint.

   • PrivateLink service name: The name of the PrivateLink service.

     The service name is the one noted above, in the last step of the Step 5 section.

   • Create an endpoint with high availability: Check the box if you wish to deploy an endpoint with High Availability.

     Endpoints deployed with high availability have network interfaces deployed in multiple availability zones.

5. Click Create to create the PrivateLink Endpoint.

6. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and then click Finish.

Step 8. Accept the endpoint connection request

1. In the AWS console, when the PrivateLink Endpoint status changes to “Pending accept”,

2. In the AWS console, in the Endpoint connections tab of the endpoint service, select the associated Endpoint ID.

3. Click Actions, and Accept. Type in accept and click Accept.

   

4. The status of the PrivateLink Endpoint will transition to “Ready” state.

Step 9. (Optional) Create the DNS record

1. When the PrivateLink Endpoint status transitions to “Ready”, click Create DNS record in the DNS tab, or click Create Record on the associated PrivateLink Endpoint tile.
2. Specify the following:
   • Egress PrivateLink Endpoint: The Egress PrivateLink Endpoint you created in the Step 7 section.
   • Domain: Associated service endpoint (<service>.<region>.amazonaws.com).
3. Click Save to create the record.

Step 10. Create the Connector

1. When the DNS Record status transitions to “Ready”, you can create the connector.

2. For the steps to create the connector, refer to the connector-specific documentation that is listed for your specific connector in Supported connectors.

3. If no DNS record was created, you will need to use the VPC endpoint DNS name for the connector endpoint configuration (i.e. hostname).

   

Troubleshooting

Issue: Connector is unable to connect to the target system even with the Egress PrivateLink Endpoint in the Ready state

Possible cause: There could be a zonal mismatch.

Solution: Enable cross-zone load balancing on your load balancer and try again.

1. In the EC2 Dashboard, browse to Load Balancing → Load Balancers, and click your load balancer.

2. Click Actions, and select Edit load balancer attributes.

3. In Availability Zone routing configuration, select Enable cross-zone balancing.