Amazon S3 Sink Connector for Confluent Cloud with AWS Egress Access Point

This topic presents the steps for configuring the Amazon S3 Sink connector in the Confluent Cloud private networking with AWS PrivateLink and Egress Access Point.

The following is a list of prerequisites for configuring the Amazon S3 Sink connector with an Egress Access Point:

  • A Confluent Cloud Dedicated cluster was set up and running within an AWS PrivateLink network.
  • A source topic was created to sink data into the S3 bucket.
  • S3 bucket was set up and running within the same region and cloud as the Confluent Cloud cluster.

Note

For additional security, you can add a S3 bucket policy to restrict access to only Confluent’s VPC Endpoint ID associated with the Egress Access Point. For details, see Restricting access to a specific VPC endpoint.

To configure the S3 Sink connector with the AWS PrivateLink and Egress Access Points:

  1. Create the Egress Access Point in Confluent Cloud.

    1. In the Confluent Cloud Console, go to Environment → Network, and select the associated Privatelink network you want to use.

    2. In the Egress Access Points tab, click Create access point.

    3. Specify the following, and click Save.

      • Name: The name for the Access point
      • PrivateLink service name: Your S3 service name (com.amazonaws.<region>.s3)
      • Create an access point with high availability: Select for high availability setup.
      ../../_images/s3-access-point.png

  2. Retrieve the VPC endpoint DNS name.

    1. Wait for the Access Point is successfully provisioned when the status changes to “Ready”.
    2. Note down the VPC endpoint DNS name to be used for S3 connector setup.
    ../../_images/s3-endpoint-dns-name.png

  3. Create the S3 Sink Connector.

    1. Go to your associated Dedicated cluster.

    2. In the left navigation menu, click Connectors.

    3. Click the Amazon S3 Sink connector card.

    4. Select the source topic.

    5. Specify the Kafka authentication mechanism.

    6. Specify the authentication details for S3.

      • Store URL: Replace * with bucket in the VPC endpoint DNS name that you retrieved when you created the Egress Access Endpoint in the previous step, and specify that value.

        For example, if the VPC endpoint DNS name is *.vpce-02ab7be7f6f5cc718-cgz9wshg.s3.us-east-2.vpce.amazonaws.com, specify https://bucket.vpce-02ab7be7f6f5cc718-cgz9wshg.s3.us-east-2.vpce.amazonaws.com.

    ../../_images/s3-store-url.png
    1. Specify configuration details for the connector.
    2. Specify sizing (number of tasks) for the connector.
    3. Review and launch the connector.

  4. When the connector is successfully launched, the connector status becomes “Running”.

Related content