Configure an AWS PrivateLink connection to Confluent Cloud
Follow this procedure to configure AWS PrivateLink for a Dedicated cluster in
A Dedicated Kafka cluster in AWS with AWS
PrivateLink enabled. For more information about how to create a dedicated
cluster, see Create a Cluster in Confluent Cloud.
The following AWS regions do not yet support AWS PrivateLink
clusters in Confluent Cloud: ca-central-1 (Canada), eu-west-2 (London),
eu-west-3 (Paris), and sa-east-1 (Sao Paulo). Instead, you can
use VPC peering for clusters
in these regions, or use AWS PrivateLink with clusters in different
Register your AWS account with Confluent Cloud
To make an AWS PrivateLink connection to a cluster in Confluent Cloud you must register the
AWS account ID you wish to use. This is a security measure so Confluent can ensure
only your organization can initiate AWS PrivateLink connections to the cluster.
AWS PrivateLink connections from a VPC not contained in a registered AWS account
will not be accepted by Confluent Cloud.
- Navigate to the Cluster Settings page, click the Networking tab, and
click Add Connection.
- Provide the 12-digit AWS Account Number for the account containing the VPCs
you want to make the PrivateLink connection from and click Save.
Your AWS PrivateLink connection status will transition from “Pending” to
“Active” in the Confluent Cloud web UI. You still need to configure the VPC endpoint
in your VPC before you can connect to the cluster.
Set up the VPC Endpoint for AWS PrivateLink in your AWS account
After the connection status is “Active” in the Confluent Cloud web UI,
you must configure a VPC Endpoint in your VPC to make the PrivateLink connection
to your Confluent Cloud cluster.
Confluent recommends using a Terraform config
for setting up the VPC Endpoint. This config automates the manual steps described below.
In the Confluent Cloud UI you will find the following information for your Confluent Cloud
cluster under the Cluster Settings section.
- Kafka Bootstrap (in the General tab)
- Availability Zone IDs (in the Networking tab)
- VPC Service Endpoint Name (in the Networking tab)
- DNS Domain (in the Networking tab)
Verify subnet availability
The Confluent Cloud VPC and cluster is created in specific zones that, for
optimal usage, should match the zones of the customer VPC you want to make
the AWS PrivateLink connection from. You must have subnets in your VPC for these
zones so that IPs can be allocated from them. It is allowed to also have subnets in zones
outside of these. AWS Zone IDs should be used for this. You can find the
specific Availability Zones for your Confluent Cloud cluster in the UI.
Ensure the VPC settings enableDnsHostnames and enableDnsSupport
are set to
Please note: Availability Zone names (like us-west-2a) are not
consistent across AWS accounts, so Availability Zone IDs (like usw2-az1)
are used instead.
Create the VPC Endpoint
In the AWS VPC Console:
- Select Endpoints from left hand list of tabs.
- Click Create Endpoint.
- Select Find service by name.
- Paste in the Confluent Cloud VPC Service Endpoint Name. You can find this in
the Confluent Cloud UI.
- Click Verify. If you get an error, ensure that your account is allowed
to create PrivateLink connections.
- Select VPC to create endpoints. Keep a note of this VPC Endpoint ID
for use later.
- All zones the service is available in should be pre-selected. Ensure
the desired subnet is selected for each zone. Confirm they match the availability
zones for Confluent Cloud that are displayed in the Confluent Cloud UI. Failure to add
all zones can cause connectivity issues to brokers in the omitted zones,
which can result in an unusable cluster.
- Select or create a security group for the VPC Endpoints. Add three inbound
rules for each of ports 80, 443, and 9092 from your desired source
(i.e. your VPC CIDR). The Protocol should be
TCP for all three rules.
Note: port 80 is not required, but is available as a
redirect only to https/443 if desired.
- Wait for acceptance by Confluent Cloud. This should happen almost immediately
(less than 1 minute). After it is accepted, the endpoint will transition from
“Pending” to “Active”.
Update the DNS configuration.
DNS changes must be made to ensure connectivity passes through AWS PrivateLink
in the supported pattern. Any DNS provider can be used - AWS Route53 (used
in this example) is not required. Any DNS provider that can ensure DNS is routed
as follows is acceptable.
To use AWS PrivateLink with Confluent Cloud, your VPC must allow outbound
internet connections for DNS resolution. Confluent Cloud Schema Registry is also only accessible
over the internet. If you’re using the Confluent Cloud CLI, it requires internet
access to authenticate with the Confluent Cloud control plane.
Run the DNS helper script
to figure out the DNS Zone records for a specific VPC Endpoint.
Update DNS using AWS Route53 in the AWS console:
Click Create Hosted Zone.
Paste Confluent Cloud DNS into Domain Name. This can be found in the
Confluent Cloud UI.
Change Type to Private Hosted Zone for Amazon VPC.
Select the VPC ID where you added the VPC Endpoint.
Create the following records with the Create Record Set button
using the VPC Endpoint DNS Name map from the previous setup in the form
*.$domain CNAME “All Zones VPC Endpoint” TTL 60
*.l92v4.us-west-2.aws.confluent.cloud CNAME vpce-09f9f4e9a86682eed-9gxp2f7v.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60
The CNAME is used to ensure AWS Route53 health checks are utilized in the case of AWS outages.
Create one record per zone (repeat for all zones) in the form
*.$zoneid.$domain CNAME “VPC Endpoint Zone” TTL 60
*.usw2-az3.l92v4.us-west-2.aws.confluent.cloud. CNAME vpce-09f9f4e9a86682eed-9gxp2f7v-us-west-2a.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60
*.usw2-az2.l92v4.us-west-2.aws.confluent.cloud. CNAME vpce-09f9f4e9a86682eed-9gxp2f7v-us-west-2c.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60
*.usw2-az1.l92v4.us-west-2.aws.confluent.cloud. CNAME vpce-09f9f4e9a86682eed-9gxp2f7v-us-west-2b.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60
Validate Connectivity to Confluent Cloud
From an instance within the VPC (or anywhere the previous step’s DNS is
set up), run the following to validate Kafka connectivity through AWS PrivateLink
is working correctly.
Set a variable with the cluster bootstrap URL.
% export BOOTSTRAP=$ConfluentCloudBootstrap
% export BOOTSTRAP=lkc-nkodz-0l6je.us-west-2.aws.confluent.cloud
Test connectivity to the cluster.
% openssl s_client -connect $BOOTSTRAP:9092 -servername $BOOTSTRAP -verify_hostname $BOOTSTRAP </dev/null 2>/dev/null | grep -E 'Verify return code|BEGIN CERTIFICATE' | xargs
If the return output is
-----BEGIN CERTIFICATE----- Verify return code: 0 (ok),
connectivity to the bootstrap is confirmed.
You might need to update the network security tools and firewalls to allow
connectivity. If you have issues connecting after following these
steps, confirm which network security systems your organization
uses and whether their configurations need to be changed. If you still have issues,
run the debug connectivity script
and provide the output to Confluent Cloud Support for assistance with your PrivateLink setup.
Next, verify connectivity with the Confluent Cloud CLI.
Log in to the Confluent Cloud CLI with your Confluent Cloud credentials.
List the clusters in your organization.
ccloud kafka cluster list
Select the cluster with AWS PrivateLink you wish to test.
ccloud kafka cluster use ...
ccloud kafka cluster use lkc-a1b2c
Create a cluster API key to authenticate with the cluster.
ccloud api-key create --resource ... --description ...
ccloud api-key create --resource lkc-a1b2c --description "connectivity test"
Select the API key you just created.
ccloud api-key use ... --resource ...
ccloud api-key use WQDMCIQWLJDGYR5Q --resource lkc-a1b2c
Create a test topic.
ccloud kafka topic create test
Start consuming events from the test topic.
ccloud kafka topic consume test
Open another terminal tab or window.
Start a producer.
ccloud kafka topic produce test
Type anything into the produce tab and hit
Ctrl+C to stop the producer.
The tab running consume will print what was typed in the tab running produce.
You’re done! The cluster is ready for use.