Egress PrivateLink Endpoint Setup: S3 on AWS for Confluent Cloud

This topic presents the steps for configuring the Amazon S3 Sink connector in Confluent Cloud with AWS PrivateLink and Egress PrivateLink Endpoint.

Prerequisites

The following is a list of prerequisites for configuring the Amazon S3 Sink connector with an Egress PrivateLink Endpoint:

• A Confluent Cloud Dedicated cluster was set up and is running within an AWS PrivateLink network.
• A source topic was created to sink data into the S3 bucket.
• S3 bucket was set up and running within the same region and cloud as the Confluent Cloud cluster.

Note

For additional security, you can add a S3 bucket policy to restrict access to only Confluent’s VPC Endpoint ID associated with the Egress PrivateLink Endpoint. For details, see Restricting access to a specific VPC endpoint.

Step 1. Create a gateway in Confluent Cloud (for Enterprise cluster only)

If using an Enterprise cluster in Confluent Cloud, you must set up a gateway in Confluent Cloud as described in Create a gateway for outbound connectivity in Confluent Cloud.

Step 2. Create an Egress PrivateLink Endpoint

1. In the Network Management tab of the desired Confluent Cloud environment, click the Confluent Cloud network you want to add the PrivateLink Endpoint to. The Connection Type of the network needs to be “PrivateLink Access”.

2. Click Create endpoint in the Egress connections tab.

3. Click the service you want to connect to, specifically, S3.

4. Follow the guided steps to specify the field values, including:

   • Name: Name of the PrivateLink Endpoint.

   • PrivateLink service name: The name of the PrivateLink service you retrieved in Obtain AWS PrivateLink Service name.

   • Create an endpoint with high availability: Check the box if you wish to deploy an endpoint with High Availability.

     Endpoints deployed with high availability have network interfaces deployed in multiple availability zones.

5. Click Create to create the PrivateLink Endpoint.

6. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and then click Finish.

7. Retrieve the VPC endpoint DNS name.

   1. Wait for the PrivateLink Endpoint is successfully provisioned when the status changes to “Ready”.
   2. Note down the VPC endpoint DNS name to be used for S3 connector setup.
   

Step 3. Create the S3 Sink connector

1. Specify the authentication details for S3.

   • Store URL: Replace * with bucket in the VPC endpoint DNS name that you retrieved when you created the Egress Private Link Endpoint in the previous step, and specify that value.

     For example, if the VPC endpoint DNS name is *.vpce-02ab7be7f6f5cc718-cgz9wshg.s3.us-east-2.vpce.amazonaws.com, specify https://bucket.vpce-02ab7be7f6f5cc718-cgz9wshg.s3.us-east-2.vpce.amazonaws.com.

   

2. Follow the steps to create the source connector or the sink connector in Confluent Cloud.

Related content

• AWS services that integrate with AWS PrivateLink