# Use AWS PrivateLink for Dedicated Clusters on Confluent Cloud [AWS PrivateLink](https://aws.amazon.com/privatelink/) allows for one-way secure connection access from your VPC to Confluent Cloud, available through [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-g5ujul6iovvcy?trk=14575e70-1766-4f20-8083-0c2757a1ec75&sc_channel=el) or [directly from Confluent](https://www.confluent.io/get-started/), with added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity. The following diagram summarizes the AWS PrivateLink architecture with the customer VPC/account and the Confluent Cloud VPC/account.  To set up to use AWS PrivateLink with Confluent Cloud: 1. Identify a Confluent Cloud network you want to use, or [set up a new Confluent Cloud network](../ccloud-network/aws.md#create-ccloud-network-aws). 2. [Add a PrivateLink Access in Confluent Cloud](#aws-privatelink-register). 3. [Provision PrivateLink endpoints in AWS](#aws-privatelink-create-connection). 4. [Set up DNS records in AWS](#aws-privatelink-dns-records). 5. [Test private link connectivity to Confluent Cloud](../testing.md#aws-privatelink-validate). 6. [Troubleshoot broker connectivity issues](../testing.md#aws-pl-troubleshoot) if necessary. The following tutorial video walks you through the process of configuring an AWS PrivateLink in AWS. It also covers common mistakes and gotchas that you will find useful when setting up an AWS PrivateLink with Confluent Cloud. ## Requirements and considerations Review the following requirements and considerations before you set up a PrivateLink in AWS with Confluent Cloud: * The AWS PrivateLink described in this document is only available for use with Dedicated clusters. For use with Enterprise clusters, see [Use AWS PrivateLink for Serverless Products on Confluent Cloud](../aws-platt.md#cloud-networking-privatelink-aws-esku). * Have a [Confluent Cloud network](../overview.md#ccloud-network-overview) of type `PrivateLink` for AWS available. If you do not have a Confluent Cloud network, see [Create Confluent Cloud Network on AWS](../ccloud-network/aws.md#create-ccloud-network-aws). * Your VPC must allow outbound internet connections for Confluent Cloud Schema Registry, ksqlDB, and Confluent CLI to work. * Confluent Cloud Schema Registry is accessible over the internet. * Provisioning new ksqlDB clusters requires Internet access. After ksqlDB clusters are up and running, they are fully accessible over AWS PrivateLink connections. * Confluent CLI requires internet access to authenticate with the Confluent Cloud control plane. * Confluent Cloud Console components, such as topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with AWS PrivateLink, see [Use the Confluent Cloud Console with Private Networking](../ccloud-console-access.md#ccloud-console-access-networking). * Existing Confluent Cloud networks cannot be converted to use AWS PrivateLink. * After provisioning of a Confluent Cloud network, you cannot change the DNS resolution option for the public or private DNS resolution. * Cross-region AWS PrivateLink Attachment Connections are not supported. * All AWS availability zones, except `use1-az3` in the `us-east-1` region, are supported. ### Connectors [Fully managed Confluent Cloud connectors](../../connectors/overview.md#kafka-connect-cloud) can connect to sources or sinks using public IP addresses. Fully managed Confluent Cloud connectors can use Egress PrivateLink Endpoints to connect to the sources or sinks in the customer network with private IP addresses. For information about configuring an Egress PrivateLink Endpoint, see [Use AWS Egress PrivateLink Endpoints for Dedicated Clusters on Confluent Cloud](../aws-egress-privatelink.md#cloud-networking-privatelink-aws-egress). ### Single availability-zone clusters Each Confluent Cloud single-zone cluster that uses AWS PrivateLink access is provisioned with service endpoints in one availability zone. The availability zone is selected based on Confluent Cloud placement policies. To ensure connectivity over AWS PrivateLink connections, provision subnets in your VPC that minimally include the single availability zone in which the AWS PrivateLink access is provisioned. ### Access to Confluent Cloud serverless products Connections established for use with Dedicated Kafka clusters may also be used to connect to some serverless products. For service-specific information, see: * [Flink](../../flink/concepts/flink-private-networking.md#flink-sql-private-networking) * [Schema Registry](../../sr/fundamentals/schema-registry-vpc.md#sr-ccloud-vpc) ## Add a PrivateLink Access in Confluent Cloud To make an AWS PrivateLink connection to a cluster in Confluent Cloud, you must register the AWS Account ID you wish to use by creating a PrivateLink access service. This is a security measure so Confluent can ensure only your organization can initiate AWS PrivateLink connections to the cluster. AWS PrivateLink connections from a VPC not contained in a registered AWS account will not be accepted by Confluent Cloud. You can register multiple AWS accounts to the same Confluent Cloud cluster, and AWS PrivateLink connections can be made from multiple VPCs in each registered AWS account. If a VPC exists in a different AWS account, you need to create a separate PrivateLink Access on your Confluent Cloud network. ### Confluent Cloud Console 1. In the **Network Management** tab of the desired Confluent Cloud environment, click the **For dedicated cluster** tab. 2. Click the Confluent Cloud network to which you want to add the connection. 3. In the **Ingress connections** tab, click **+ PrivateLink Access**. 4. Specify the following field values. * **Name**: The name for this PrivateLink Access. * **AWS Account Number**: Enter the 12-digit **AWS Account Number** for the account containing the VPCs you want to make the AWS PrivateLink connection from. 5. Note the **VPC Endpoint service name** to create an AWS PrivateLink connection from your VPC to the Confluent Cloud cluster. This URL will also be provided later.  6. Click **Add**, and wait for the AWS PrivateLink connection status to transition from **Pending** to **Ready** before you create a Private Endpoint in AWS. ### Confluent REST API **HTTP POST request** ```text POST https://api.confluent.cloud/networking/v1/private-link-accesses ``` **Authentication** See [Authentication](https://docs.confluent.io/cloud/current/api.html/#authentication). **Request specification** In the request specification, include values for the Confluent Cloud network ID, account, environment, and, optionally, add the display name. Update the attributes below with the correct values. ```json { "spec": { "display_name": "AWS-PL-CCN-1", "cloud": { "kind": "AwsPrivateLinkAccess", "account": "000000000000" }, "environment": { "id":"env-abc123" }, "network": { "id":"n-00000" } } } ``` ### Confluent CLI Use the [confluent network private-link access create](https://docs.confluent.io/confluent-cli/current/command-reference/overview.html) Confluent CLI command to create an AWS private link access: ```bash confluent network private-link access create ``` The following are the command-specific flags: * `--network`: Required. Confluent Cloud network ID. * `--cloud`: Required. The cloud provider. Set to `aws`. * `--cloud-account`. Required. The AWS account ID for the account containing the VPCs you want to connect from using AWS PrivateLink. You can specify additional optional CLI flags described in the [Confluent CLI command reference](https://docs.confluent.io/confluent-cli/current/command-reference/overview.html), such as `--environment`. The following is an example Confluent CLI command to create a private link access: ```bash confluent network private-link access create my-private-link-access \ --network n-123456 \ --cloud aws \ --cloud-account 123456789012 ``` ### Terraform Use the [confluent_private_link_access](https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_private_link_access) Confluent Terraform Provider resource to create a PrivateLink Access. An example snippet of Terraform configuration: ```terraform resource "confluent_environment" "development" { display_name = "Development" } resource "confluent_network" "aws-private-link" { display_name = "AWS Private Link Network" cloud = "AWS" region = "us-east-1" connection_types = ["PRIVATELINK"] zones = ["use1-az1", "use1-az2", "use1-az6"] environment { id = confluent_environment.development.id } lifecycle { prevent_destroy = true } } resource "confluent_private_link_access" "aws" { display_name = "AWS Private Link Access" aws { account = "012345678901" } environment { id = confluent_environment.development.id } network { id = confluent_network.aws-private-link.id } lifecycle { prevent_destroy = true } } ``` See more Terraform configuration examples for creating a PrivateLink connection using Terraform: * [PrivateLink with ACLS](https://github.com/confluentinc/terraform-provider-confluent/tree/master/examples/configurations/dedicated-privatelink-aws-kafka-acls) * [PrivateLink with RBAC](https://github.com/confluentinc/terraform-provider-confluent/tree/master/examples/configurations/dedicated-privatelink-aws-kafka-rbac)
When the AWS PrivateLink connection status transitions from **Pending** to **Ready** in the Confluent Cloud Console, proceed to configure the Private Endpoints in your VPC. ## Provision PrivateLink endpoints in AWS When the connection status becomes **Ready** in the Confluent Cloud Console, provision a VPC private endpoint in the [AWS VPC dashboard](https://console.aws.amazon.com/vpc/home) to make the AWS PrivateLink connection to your Confluent Cloud cluster. For the current process of creating a VPC private endpoint, refer to [Create a VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#create-interface-endpoint-aws). 1. In the Confluent Cloud Console, find and make a note of the following information for your Confluent Cloud cluster under the **Cluster Settings** section and Confluent Cloud network under **Confluent Cloud Network overview**: * In **Cluster Settings**: * Bootstrap server endpoint in the **Endpoints** section * Zones in the **Cloud details** section These are the availability zones of the Kafka cluster. * In **Networking** > **Details**: * Zones: Availability zone IDs * VPC Endpoint service name * DNS domain * DNS subdomain 2. Open the [AWS Management Console](https://console.aws.amazon.com/vpc/home) and browse to the VPC you want to use for the PrivateLink connection. 3. Verify subnet availability in your AWS VPC, and confirm the selected subnets match the availability zones for Confluent Cloud that you noted in the first step. #### IMPORTANT The zones for the Confluent Cloud VPC and cluster must match the zones of the VPC you want to make the AWS PrivateLink connections from. Have the matching subnets in your VPC for these zones so that IP addresses can be allocated from them. 4. Verify that **Enable DNS resolution** and **Enable DNS hostnames** are enabled. If the settings are not enabled, click **Actions** > **VPC settings**, and enable the **Enable DNS resolution** and **Enable DNS hostnames** settings. 5. Create or edit a security group you want to use for the new VPC endpoint. - Add three inbound rules for each of ports `80`, `443`, and `9092` from your desired source (your VPC CIDR). The **Protocol** should be `TCP` for all three rules. - Port `80` is not required, but is available as a redirect only to `https/443`, if desired. 6. Create a VPC endpoint. 1. In the navigation menu under **VIRTUAL PRIVATE CLOUD**, click **Endpoints**. 2. Click **Create endpoint**, and specify the following settings for the endpoint: * **Service category**: Select **PrivateLink Ready partner services**. * **Service settings**: Enter the **Service name** for your Confluent Cloud **VPC Endpoint service name**. You can find this in the Confluent Cloud Console (as described in Step #1 in this section). Click **Verify service**. If you get an error, ensure that your account is allowed to create PrivateLink connections. * **VPC**: Select the **VPC** in which to create your endpoint. * **Subnets**: Select the subnet for the availability zones for your Confluent Cloud network. Ensure that the desired subnet is selected for each zone. Confirm that the selected subnets match the availability zones in the **Networking** tab in the Confluent Cloud Console (as described in Step #1). Failure to add all zones as displayed in the Confluent Cloud Console can cause connectivity issues to brokers in the omitted zones, which can result in an unusable cluster. Confluent Cloud single availability zone clusters need service and subnet selection in one zone whereas Confluent Cloud multi-availability zone clusters need service and subnet selection in three zones. * **Enable DNS name** setting under **Additional settings** (only appearing after the VPC is selected): Uncheck. This step is required for all types of DNS resolution for Confluent Cloud. * **Security groups**: Select the security group that you previously created or edited. 3. Click **Create endpoint**. Your VPC endpoint is created and displayed. Copy the VPC Endpoint ID for later use. 4. When Confluent Cloud accepts the VPC endpoint connection, the endpoint transitions from **Pending** to **Ready**. On the AWS console, you can see the VPC endpoint, the subnets, and the DNS names associated with the VPC Endpoint. ## Set up DNS records in AWS You must create your DNS records to ensure connectivity through AWS PrivateLink in the supported pattern. You can use any DNS service that routes DNS requests as described in this section. AWS Route 53 is used for the example in this section, ### DNS resolution options For AWS PrivateLink Confluent Cloud networks, you can use the public or private DNS resolution: * The private DNS resolution is the recommended option and guarantees fully private DNS resolution. * The public DNS resolution is useful when you want to ensure that Confluent deployments are homogenous and conform to DNS configurations for your networks. DNS resolution is selected when you create a Confluent Cloud network, and it cannot be modified after creating the Confluent Cloud network. See [Create a Confluent Cloud network](../ccloud-network/aws.md#ccn-config-aws). #### Public DNS resolution The public (also known as chased private in Confluent Cloud) DNS resolution is used for the bootstrap server and broker hostnames of a Confluent Cloud cluster that is using AWS Private Link. When the public resolution is used, the clusters in this network require both public and private DNS to resolve cluster endpoints. Only the Confluent Global DNS Resolver (GLB) endpoints are advertised. The public DNS resolution performs the following two-step process: 1. The Confluent Cloud Global DNS Resolver removes the `glb` subdomain and returns a CNAME for your bootstrap and broker hostnames. Example: `$lkc-id-$nid.$region.$cloud.glb.confluent.cloud` CNAME returned: `$lkc-id.$nid.$region.$cloud.confluent.cloud` 2. The CNAME resolves to your VPC private endpoints based on the Private Hosted Zone configuration. #### Private DNS resolution When the private DNS resolution is used, the clusters in this network only require private DNS to resolve cluster endpoints. Only non-GLB endpoints are advertised. #### NOTE The private DNS in Confluent Cloud does not refer to the [AWS private DNS name feature](https://docs.aws.amazon.com/vpc/latest/privatelink/manage-dns-names.html). ### Create a DNS zone and DNS records DNS entries need to be created for PrivateLink irrespective of the DNS resolution option you selected when creating the Confluent Cloud network. These DNS records map the Confluent Cloud DNS names to the AWS VPC DNS names. To create DNS entries using AWS Route 53: 1. Browse to Route 53 in the AWS Management Console. 2. Create the private hosted zone for the PrivateLink. 1. In **Hosted zones**, click **Create hosted zone**. * **Domain name**: Specify the Confluent Cloud DNS domain name from the **Networking** tab under **Cluster Settings** in the Confluent Cloud Console. * **Type**: Select **Private hosted zone**. * **Region**: Select the region of the PrivateLink. * **VPC ID**: Select the VPC ID where you added the VPC endpoint. 2. Click **Create hosted zone**. 3. (Optional) To figure out the correct DNS zone records for specific VPC endpoints for Confluent Cloud, you can run the [DNS helper shell script](https://github.com/confluentinc/ccloud-connectivity/blob/master/privatelink/aws/dns-endpoints.sh) from an EC2 instance within the VPC. You specify the endpoint DNS name as an argument. The output contains the domain names, the record types, and the DNS values that you can input when you create DNS records. For example: ```bash ./dns-endpoint.sh vpce-01703e07518894776 * CNAME vpce-01703e07518894776-yjj63e02.vpce-svc-00059a202ddac4cea.eu-west-1.vpce.amazonaws.com *.euw1-az1 CNAME vpce-01703e07518894776-yjj63e02-eu-west-1a.vpce-svc-00059a202ddac4cea.eu-west-1.vpce.amazonaws.com *.euw1-az3 CNAME vpce-01703e07518894776-yjj63e02-eu-west-1c.vpce-svc-00059a202ddac4cea.eu-west-1.vpce.amazonaws.com *.euw1-az2 CNAME vpce-01703e07518894776-yjj63e02-eu-west-1b.vpce-svc-00059a202ddac4cea.eu-west-1.vpce.amazonaws.com ``` 4. Add DNS records. You can get the endpoint DNS names from the AWS console under the endpoint created earlier. The hosted zone will contain four CNAME DNS records, for one bootstrap server endpoint and three zonal endpoints. 1. In the new DNS hosted zone, click **Create record** to create a DNS record for bootstrap endpoint resolution. * **Record name**: `*` The field will already have the VPC Endpoint DNS domain name (the bootstrap server endpoint) in Confluent Cloud. An example record name: `*.l92v4.us-west-2.aws.confluent.cloud` * **Record type**: `CNAME` * **Value**: The DNS name of the VPC endpoint you created in [Provision PrivateLink endpoints in AWS](#aws-privatelink-create-connection). You can look up the DNS name on the Endpoint detail page as shown in the image below. In the **DNS names** section, the DNS name of the VPC endpoint is listed, followed by the DNS names of the zonal endpoints.  * **TTL**: `60` 2. Create three zonal DNS records, one zonal DNS record per availability zone of the Confluent Cloud network. * **Record name**: `*.` Use the DNS subdomain ID you retrieved in [Provision PrivateLink endpoints in AWS](#aws-privatelink-create-connection). The field will already have the Confluent Cloud VPC endpoint DNS domain name. An example record name: `*.usw2-az3.l92v4.us-west-2.aws.confluent.cloud` * **Record type**: `CNAME` * **Value**: The DNS name of this availability zonal VPC endpoint you created in [Provision PrivateLink endpoints in AWS](#aws-privatelink-create-connection). The CNAME records must point to the correct subnet/zonal endpoints. This mapping must be respected in order to successfully connect to the Confluent Cloud cluster. Refer to the **Subnets** tab of the VPC endpoint to relate the availability zone IDs to the canonical name, and specify the corresponding DNS name that includes the canonical name. For example, to create a zonal DNS record for the `us-west-2a` availability zone using the Endpoint shown below, set **Record name** to the availability zone ID, `usw2-az1`. And set the DNS record **Value** to the endpoint domain that includes the corresponding availability zone, `us-west-2a`.  * **Record name**: `*.usw2-az1` * **Record type**: `CNAME` * **Value**: ```text vpce-04de653cd9cf5c1b7-3sfnkgjj-us-west-2a.vpce-svc-031ca3470fb6f14bd.us-west-2.vpce.amazonaws.com ``` 3. Click **Create records**. #### NOTE In Confluent Cloud with private linking, Kafka broker names you retrieve from the metadata are not static. Do not hardcode the broker names in DNS records. The following are example DNS record sets.  Your cluster is now ready for use. If you encounter any problem, refer to [troubleshoot connectivity issues](../testing.md#aws-pl-troubleshoot). ## Next steps Try [Confluent Cloud on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-g5ujul6iovvcy?trk=14575e70-1766-4f20-8083-0c2757a1ec75&sc_channel=el) with $1000 of free usage for 30 days, and pay as you go. No credit card is required.