Egress PrivateLink Endpoint Setup: DynamoDB CDC Source on AWS for Confluent Cloud

This topic presents the steps for configuring the Amazon DynamoDB CDC Source connector in Confluent Cloud with AWS PrivateLink and Egress PrivateLink endpoints.

The DynamoDB CDC Source connector uses two distinct AWS API endpoints during operation:

• DynamoDB API (dynamodb.<region>.amazonaws.com): Used during the snapshot phase to scan table data.

• DynamoDB Streams API (streams.dynamodb.<region>.amazonaws.com): Used during the CDC phase to capture change events. The connector constructs this URL internally based on your configured DynamoDB endpoint.

Because both endpoints must be reachable through private networking, you must create two separate Egress PrivateLink endpoints and two DNS records in Confluent Cloud.

Confluent Cloud is available through AWS Marketplace or directly from Confluent.

Prerequisites

The following are prerequisites for configuring the Amazon DynamoDB CDC Source connector with Egress PrivateLink endpoints:

• In Confluent Cloud, one of the following cluster types is set up with the specified network resource:

  • A Dedicated cluster with a Confluent Cloud network.

    For the steps to create a Confluent Cloud network, see Create a Confluent Cloud network. The Connection type of the network needs to be PrivateLink Access.

  • A Enterprise cluster with a network gateway.

    For the steps to create a gateway, see Create a gateway for outbound connectivity in Confluent Cloud.

• Amazon DynamoDB tables are in the same region as the Confluent Cloud network resource.

• The IAM policy for the connector includes all required DynamoDB permissions. For more information, see IAM policy DynamoDB CDC.

Note

On PrivateLink clusters, the DynamoDB CDC Source connector requires a Service Account for Kafka authentication. API key authentication is not supported on PrivateLink clusters.

Step 1. Obtain the DynamoDB VPC endpoint service name

The Amazon DynamoDB VPC interface endpoint service name follow formats given below:

com.amazonaws.<region>.dynamodb
com.amazonaws.<region>.dynamodb-streams

Replace <region> with your Amazon region (for example, us-east-1).

Use these service name as the PrivateLink service name when creating both Egress PrivateLink endpoints in the following steps.

For more information, see the AWS documentation on DynamoDB VPC endpoints.

Step 2. Create two Egress PrivateLink endpoints

Create two separate Egress PrivateLink endpoints: one for the DynamoDB API (snapshot) and one for the DynamoDB Streams API (CDC). Both endpoints use the respective Amazon service names obtained in Step 1. Obtain the DynamoDB VPC endpoint service name.

Dedicated cluster

Repeat the following steps twice — once for the DynamoDB endpoint and once for the DynamoDB Streams endpoint.

1. In the Network management page or tab of the desired Confluent Cloud environment, click the Confluent Cloud network you want to add the PrivateLink endpoint to. The Connection Type of the network needs to be PrivateLink Access.

2. Click Create endpoint in the Egress connections tab.

3. Click Other as the service (DynamoDB is not listed as a preset service).

4. Specify the following field values:

   • SERVICE: Enter a descriptive name for the service, for example, DynamoDB or DynamoDB Streams.

   • Endpoint Name: Enter a unique name for the PrivateLink endpoint.

   • Create an endpoint with high availability: Select this checkbox to deploy an endpoint with high availability.

   • PrivateLink service name: Enter the VPC service name retrieved in Step 1. Obtain the DynamoDB VPC endpoint service name (for example, com.amazonaws.us-east-1.dynamodb).

5. Click Create to create the PrivateLink endpoint.

6. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and then click Finish.

7. Repeat the preceding steps to create the second endpoint.

Enterprise cluster

Repeat the following steps twice — once for the DynamoDB endpoint and once for the DynamoDB Streams endpoint.

1. In the Network management page or tab of the desired Confluent Cloud environment, click the For serverless products tab.

2. Click the gateway to which you want to add the PrivateLink endpoint.

3. In the Access points tab, click Add access point.

4. Click Other as the service (DynamoDB is not listed as a preset service).

5. Specify the following field values:

   • Access point name: Enter a unique name for the PrivateLink endpoint, for example, dynamodb-snapshot or dynamodb-streams-cdc.

   • Create an endpoint with high availability: Select this checkbox to deploy an endpoint with high availability.

   • PrivateLink service name: Enter the VPC service name retrieved in Step 1. Obtain the DynamoDB VPC endpoint service name (for example, com.amazonaws.us-east-1.dynamodb).

6. Click Create access point to create the PrivateLink endpoint.

7. If there are additional steps for the specific target service, follow the prompt to complete the tasks, and then click Finish.

8. Repeat the preceding steps to create the second access point.

Step 3. Create two DNS records

Create two DNS records to route DynamoDB API and DynamoDB Streams API traffic through the respective PrivateLink endpoints. DNS records must be created after the endpoints transition to Ready status.

Dedicated cluster

Repeat the following steps twice — once using the DynamoDB endpoint and domain, and once using the DynamoDB Streams endpoint and domain.

1. When the PrivateLink endpoint status transitions to Ready, in the DNS tab, click Create record on the associated PrivateLink endpoint.

2. For the DynamoDB API DNS record, specify the following field values and click Save:

   • Access point: Select the DynamoDB PrivateLink endpoint created in Step 2. Create two Egress PrivateLink endpoints.

   • Domain: Enter dynamodb.<region>.amazonaws.com, replacing <region> with your Amazon region (for example, dynamodb.us-east-1.amazonaws.com).

3. For the DynamoDB Streams API DNS record, specify the following field values and click Save:

   • Access point: Select the DynamoDB Streams PrivateLink endpoint created in Step 2. Create two Egress PrivateLink endpoints.

   • Domain: Enter streams.dynamodb.<region>.amazonaws.com, replacing <region> with your Amazon region (for example, streams.dynamodb.us-east-1.amazonaws.com).

Enterprise cluster

Repeat the following steps twice — once using the DynamoDB endpoint and domain, and once using the DynamoDB Streams endpoint and domain.

1. In the Network Management tab of your environment, click the For serverless products tab, and click the Confluent Cloud gateway.

2. In the DNS tab, click Create DNS record.

3. For the DynamoDB API DNS record, specify the following field values and click Save:

   • Access point: Select the DynamoDB PrivateLink access point created in Step 2. Create two Egress PrivateLink endpoints.

   • Domain: Enter dynamodb.<region>.amazonaws.com, replacing <region> with your Amazon region (for example, dynamodb.us-east-1.amazonaws.com).

4. For the DynamoDB Streams API DNS record, specify the following field values and click Save:

   • Access point: Select the DynamoDB Streams PrivateLink access point created in Step 2. Create two Egress PrivateLink endpoints.

   • Domain: Enter streams.dynamodb.<region>.amazonaws.com, replacing <region> with your Amazon region (for example, streams.dynamodb.us-east-1.amazonaws.com).

Step 4. Create the DynamoDB CDC Source connector

1. Create the Amazon DynamoDB CDC Source connector on your PrivateLink cluster. For the connector setup steps, see Amazon DynamoDB CDC Source Connector for Confluent Cloud.

2. When configuring the connector authentication, use a Service Account for Kafka authentication. API key authentication is not supported on PrivateLink clusters.

3. On the Authentication step, configure the DynamoDB service endpoint using the standard Amazon regional endpoint URL for your region:

   https://dynamodb.<region>.amazonaws.com
   

   For example:

   https://dynamodb.us-east-1.amazonaws.com
   

   This URL resolves to the DynamoDB DNS record created in the previous step. The connector automatically constructs the DynamoDB Streams endpoint (streams.dynamodb.<region>.amazonaws.com) for CDC, which resolves through the second DNS record.