Use AWS PrivateLink for Dedicated Clusters on Confluent Cloud

AWS PrivateLink allows for one-way secure connection access from your VPC to Confluent Cloud, available through AWS Marketplace or directly from Confluent, with added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity.

The following diagram summarizes the AWS PrivateLink architecture with the customer VPC/account and the Confluent Cloud VPC/account.

To set up to use AWS PrivateLink with Confluent Cloud:

1. Identify a Confluent Cloud network you want to use, or set up a new Confluent Cloud network.
2. Add a PrivateLink Access in Confluent Cloud.
3. Provision PrivateLink endpoints in AWS.
4. Set up DNS records in AWS.
5. Test private link connectivity to Confluent Cloud.
6. Troubleshoot broker connectivity issues if necessary.

The following tutorial video walks you through the process of configuring an AWS PrivateLink in AWS. It also covers common mistakes and gotchas that you will find useful when setting up an AWS PrivateLink with Confluent Cloud.

Requirements and considerations

Review the following requirements and considerations before you set up a PrivateLink in AWS with Confluent Cloud:

• The AWS PrivateLink described in this document is only available for use with Dedicated clusters.

  For use with Enterprise clusters, see Use AWS PrivateLink for Serverless Products on Confluent Cloud.

• Have a Confluent Cloud network of type PrivateLink for AWS available. If you do not have a Confluent Cloud network, see Create Confluent Cloud Network on AWS.

• Your VPC must allow outbound internet connections for Confluent Cloud Schema Registry, ksqlDB, and Confluent CLI to work.

  • Confluent Cloud Schema Registry is accessible over the internet.
  • Provisioning new ksqlDB clusters requires Internet access. After ksqlDB clusters are up and running, they are fully accessible over AWS PrivateLink connections.
  • Confluent CLI requires internet access to authenticate with the Confluent Cloud control plane.

• Confluent Cloud Console components, such as topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with AWS PrivateLink, see Use the Confluent Cloud Console with Private Networking.

• Existing Confluent Cloud networks cannot be converted to use AWS PrivateLink.

• After provisioning of a Confluent Cloud network, you cannot change the DNS resolution option for the public or private DNS resolution.

• Cross-region AWS PrivateLink Attachment Connections are not supported.

• All AWS availability zones, except use1-az3 in the us-east-1 region, are supported.

Connectors

Fully-managed Confluent Cloud connectors can connect to sources or sinks using public IP addresses.

Fully-managed Confluent Cloud connectors can use Egress PrivateLink Endpoints to connect to the sources or sinks in the customer network with private IP addresses. For information about configuring an Egress PrivateLink Endpoint, see Use AWS Egress PrivateLink Endpoints for Dedicated Clusters on Confluent Cloud.

Single availability-zone clusters

Each Confluent Cloud single-zone cluster that uses AWS PrivateLink access is provisioned with service endpoints in one availability zone. The availability zone is selected based on Confluent Cloud placement policies.

To ensure connectivity over AWS PrivateLink connections, provision subnets in your VPC that minimally include the single availability zone in which the AWS PrivateLink access is provisioned.

Access to Confluent Cloud serverless products

Connections established for use with Dedicated Kafka clusters may also be used to connect to some serverless products. For service-specific information, see:

• Flink
• Schema Registry

Add a PrivateLink Access in Confluent Cloud

To make an AWS PrivateLink connection to a cluster in Confluent Cloud, you must register the AWS Account ID you wish to use by creating a PrivateLink access service. This is a security measure so Confluent can ensure only your organization can initiate AWS PrivateLink connections to the cluster. AWS PrivateLink connections from a VPC not contained in a registered AWS account will not be accepted by Confluent Cloud.

You can register multiple AWS accounts to the same Confluent Cloud cluster, and AWS PrivateLink connections can be made from multiple VPCs in each registered AWS account.

If a VPC exists in a different AWS account, you need to create a separate PrivateLink Access on your Confluent Cloud network.

Confluent Cloud ConsoleConfluent REST APIConfluent CLITerraform

1. In the Network Management tab of the desired Confluent Cloud environment, click the For dedicated cluster tab.

2. Click the Confluent Cloud network to which you want to add the connection.

3. In the Ingress connections tab, click + PrivateLink Access.

4. Specify the following field values.

   • Name: The name for this PrivateLink Access.
   • AWS Account Number: Enter the 12-digit AWS Account Number for the account containing the VPCs you want to make the AWS PrivateLink connection from.

5. Note the VPC Endpoint service name to create an AWS PrivateLink connection from your VPC to the Confluent Cloud cluster. This URL will also be provided later.

   

6. Click Add, and wait for the AWS PrivateLink connection status to transition from Pending to Ready before you create a Private Endpoint in AWS.

When the AWS PrivateLink connection status transitions from Pending to Ready in the Confluent Cloud Console, proceed to configure the Private Endpoints in your VPC.

Provision PrivateLink endpoints in AWS

When the connection status becomes Ready in the Confluent Cloud Console, provision a VPC private endpoint in the AWS VPC dashboard to make the AWS PrivateLink connection to your Confluent Cloud cluster.

For the current process of creating a VPC private endpoint, refer to Create a VPC endpoint.

1. In the Confluent Cloud Console, find and make a note of the following information for your Confluent Cloud cluster under the Cluster Settings section and Confluent Cloud network under Confluent Cloud Network overview:

   • In Cluster Settings:

     • Bootstrap server endpoint in the Endpoints section

     • Zones in the Cloud details section

       These are the availability zones of the Kafka cluster.

   • In Networking > Details:

     • Zones: Availability zone IDs
     • VPC Endpoint service name
     • DNS domain
     • DNS subdomain

2. Open the AWS Management Console and browse to the VPC you want to use for the PrivateLink connection.

3. Verify subnet availability in your AWS VPC, and confirm the selected subnets match the availability zones for Confluent Cloud that you noted in the first step.

   Important

   The zones for the Confluent Cloud VPC and cluster must match the zones of the VPC you want to make the AWS PrivateLink connections from. Have the matching subnets in your VPC for these zones so that IP addresses can be allocated from them.

4. Verify that Enable DNS resolution and Enable DNS hostnames are enabled.

   If the settings are not enabled, click Actions > VPC settings, and enable the Enable DNS resolution and Enable DNS hostnames settings.

5. Create or edit a security group you want to use for the new VPC endpoint.

   • Add three inbound rules for each of ports 80, 443, and 9092 from your desired source (your VPC CIDR). The Protocol should be TCP for all three rules.
   • Port 80 is not required, but is available as a redirect only to https/443, if desired.

6. Create a VPC endpoint.

   1. In the navigation menu under VIRTUAL PRIVATE CLOUD, click Endpoints.

   2. Click Create endpoint, and specify the following settings for the endpoint:

      • Service category: Select PrivateLink Ready partner services.

      • Service settings: Enter the Service name for your Confluent Cloud VPC Endpoint service name. You can find this in the Confluent Cloud Console (as described in Step #1 in this section).

        Click Verify service. If you get an error, ensure that your account is allowed to create PrivateLink connections.

      • VPC: Select the VPC in which to create your endpoint.

      • Subnets: Select the subnet for the availability zones for your Confluent Cloud network.

        Ensure that the desired subnet is selected for each zone. Confirm that the selected subnets match the availability zones in the Networking tab in the Confluent Cloud Console (as described in Step #1). Failure to add all zones as displayed in the Confluent Cloud Console can cause connectivity issues to brokers in the omitted zones, which can result in an unusable cluster.

        Confluent Cloud single availability zone clusters need service and subnet selection in one zone whereas Confluent Cloud multi-availability zone clusters need service and subnet selection in three zones.

      • Enable DNS name setting under Additional settings (only appearing after the VPC is selected): Uncheck.

        This step is required for all types of DNS resolution for Confluent Cloud.

      • Security groups: Select the security group that you previously created or edited.

   3. Click Create endpoint.

      Your VPC endpoint is created and displayed. Copy the VPC Endpoint ID for later use.

   4. When Confluent Cloud accepts the VPC endpoint connection, the endpoint transitions from Pending to Ready.

      On the AWS console, you can see the VPC endpoint, the subnets, and the DNS names associated with the VPC Endpoint.

Set up DNS records in AWS

You must create your DNS records to ensure connectivity through AWS PrivateLink in the supported pattern. You can use any DNS service that routes DNS requests as described in this section. AWS Route 53 is used for the example in this section,

DNS resolution options

For AWS PrivateLink Confluent Cloud networks, you can use the public or private DNS resolution:

• The private DNS resolution is the recommended option and guarantees fully private DNS resolution.
• The public DNS resolution is useful when you want to ensure that Confluent deployments are homogenous and conform to DNS configurations for your networks.

DNS resolution is selected when you create a Confluent Cloud network, and it cannot be modified after creating the Confluent Cloud network. See Create a Confluent Cloud network.

Public DNS resolution

The public (also known as chased private in Confluent Cloud) DNS resolution is used for the bootstrap server and broker hostnames of a Confluent Cloud cluster that is using AWS Private Link. When the public resolution is used, the clusters in this network require both public and private DNS to resolve cluster endpoints.

Only the Confluent Global DNS Resolver (GLB) endpoints are advertised.

The public DNS resolution performs the following two-step process:

1. The Confluent Cloud Global DNS Resolver removes the glb subdomain and returns a CNAME for your bootstrap and broker hostnames.

   Example: $lkc-id-$nid.$region.$cloud.glb.confluent.cloud

   CNAME returned: $lkc-id.$nid.$region.$cloud.confluent.cloud

2. The CNAME resolves to your VPC private endpoints based on the Private Hosted Zone configuration.

Private DNS resolution

When the private DNS resolution is used, the clusters in this network only require private DNS to resolve cluster endpoints. Only non-GLB endpoints are advertised.

Note

The private DNS in Confluent Cloud does not refer to the AWS private DNS name feature.

Create a DNS zone and DNS records

DNS entries need to be created for PrivateLink irrespective of the DNS resolution option you selected when creating the Confluent Cloud network. These DNS records map the Confluent Cloud DNS names to the AWS VPC DNS names.

To create DNS entries using AWS Route 53:

1. Browse to Route 53 in the AWS Management Console.

2. Create the private hosted zone for the PrivateLink.

   1. In Hosted zones, click Create hosted zone.
      • Domain name: Specify the Confluent Cloud DNS domain name from the Networking tab under Cluster Settings in the Confluent Cloud Console.
      • Type: Select Private hosted zone.
      • Region: Select the region of the PrivateLink.
      • VPC ID: Select the VPC ID where you added the VPC endpoint.
   2. Click Create hosted zone.

3. (Optional) To figure out the correct DNS zone records for specific VPC endpoints for Confluent Cloud, you can run the DNS helper shell script from an EC2 instance within the VPC.

   You specify the endpoint DNS name as an argument.

   The output contains the domain names, the record types, and the DNS values that you can input when you create DNS records. For example:

   ./dns-endpoint.sh vpce-01703e07518894776
   
   *               CNAME vpce-01703e07518894776-yjj63e02.vpce-svc-00059a202ddac4cea.eu-west-1.vpce.amazonaws.com
   *.euw1-az1      CNAME vpce-01703e07518894776-yjj63e02-eu-west-1a.vpce-svc-00059a202ddac4cea.eu-west-1.vpce.amazonaws.com
   *.euw1-az3      CNAME vpce-01703e07518894776-yjj63e02-eu-west-1c.vpce-svc-00059a202ddac4cea.eu-west-1.vpce.amazonaws.com
   *.euw1-az2      CNAME vpce-01703e07518894776-yjj63e02-eu-west-1b.vpce-svc-00059a202ddac4cea.eu-west-1.vpce.amazonaws.com
   

4. Add DNS records.

   You can get the endpoint DNS names from the AWS console under the endpoint created earlier.

   The hosted zone will contain four CNAME DNS records, for one bootstrap server endpoint and three zonal endpoints.

   1. In the new DNS hosted zone, click Create record to create a DNS record for bootstrap endpoint resolution.

      • Record name: *

        The field will already have the VPC Endpoint DNS domain name (the bootstrap server endpoint) in Confluent Cloud.

        An example record name: *.l92v4.us-west-2.aws.confluent.cloud

      • Record type: CNAME

      • Value: The DNS name of the VPC endpoint you created in Provision PrivateLink endpoints in AWS.

        You can look up the DNS name on the Endpoint detail page as shown in the image below. In the DNS names section, the DNS name of the VPC endpoint is listed, followed by the DNS names of the zonal endpoints.

        

      • TTL: 60

   2. Create three zonal DNS records, one zonal DNS record per availability zone of the Confluent Cloud network.

      • Record name: *.<Availability Zone ID>

        Use the DNS subdomain ID you retrieved in Provision PrivateLink endpoints in AWS.

        The field will already have the Confluent Cloud VPC endpoint DNS domain name.

        An example record name: *.usw2-az3.l92v4.us-west-2.aws.confluent.cloud

      • Record type: CNAME

      • Value: The DNS name of this availability zonal VPC endpoint you created in Provision PrivateLink endpoints in AWS.

        The CNAME records must point to the correct subnet/zonal endpoints. This mapping must be respected in order to successfully connect to the Confluent Cloud cluster.

        Refer to the Subnet tab of the VPC endpoint to relate the availability zone IDs to the canonical name, and specify the corresponding DNS name that includes the canonical name.

        For example, to create a zonal DNS record for the us-west-2a availability zone using the Endpoint shown below, set Record name to the availability zone ID, usw2-az1. And set the DNS record Value to the endpoint domain that includes the corresponding availability zone, us-west-2a.

        

        • Record name: *.usw2-az1

        • Record type: CNAME

        • Value:

          vpce-04de653cd9cf5c1b7-3sfnkgjj-us-west-2a.vpce-svc-031ca3470fb6f14bd.us-west-2.vpce.amazonaws.com
          

   3. Click Create records.

   Note

   In Confluent Cloud with private linking, Kafka broker names you retrieve from the metadata are not static. Do not hardcode the broker names in DNS records.

   The following are example DNS record sets.

   

Your cluster is now ready for use. If you encounter any problem, refer to troubleshoot connectivity issues.

Next steps

Try Confluent Cloud on AWS Marketplace with $1000 of free usage for 30 days, and pay as you go. No credit card is required.