Azure Private Link
allows for one-way secure connection access from your VNet to Confluent Cloud with an
added protection against data exfiltration. This networking option is popular
for its unique combination of security and simplicity of setup.
The following diagram summarizes the Azure Private Link architecture between the
VNet or subscription and the Confluent Cloud cluster.
To use an Azure Private Link service with Confluent Cloud, your VNet must
allow outbound internet connections for DNS resolution, Confluent Cloud Schema Registry,
ksqlDB, and Confluent CLI to work.
DNS requests to public authority traversing to private DNS zone is required.
Confluent Cloud Schema Registry is accessible over the internet.
Provisioning new ksqlDB instances requires Internet access. After
ksqlDB instances are up and running, they are fully accessible over
Azure Private Link connections.
Confluent CLI requires internet access to authenticate with the
Confluent Cloud control plane.
Confluent Cloud Console components, like topic management, need
additional configuration to function as they use cluster endpoints. To use
all features of the Confluent Cloud Console with Azure Private Link, see
Configure DNS Resolution.
For limitations of the Azure Private Link, see Limitations
In the request specification, include values for cloud, region, environment, connection
type, and, optionally, add the display name, CIDR, and zones for the Confluent Cloud network.
Update the attributes below with the correct values.
Register your Azure subscription with Confluent Cloud¶
Register your Azure subscription with the Confluent Cloud network for automatic approval of
private endpoint connections to the Confluent Cloud network. If required, you can
register multiple subscriptions.
In the Confluent Cloud Console, go to your network resource in the
Network Management tab and click + Private Link.
Enter the Azure subscription ID for the account containing the VNets
you want to make the Azure Private Link connection from. The Azure
subscription number can be found on your Azure subscription page of
the Azure Portal.
HTTP POST request
Your Azure Private Link connection status will transition from “Pending” to
“Active” in the Confluent Cloud Console. You still need to configure the Private
Endpoints in your VNet before you can connect to the cluster.
Note the Private Link Service Endpoint to create an Azure Private Link
connection from your VNet to the Confluent Cloud cluster. This URL will also be provided
Create an Azure Private Link connection to Confluent Cloud¶
Follow this procedure to create an Azure Private Link connection to a Confluent Cloud
cluster on Azure using the Confluent Cloud Console or REST APIs.
Set up the VNet Endpoint for Azure Private Link in your Azure account¶
After the connection status is “Active” in the Confluent Cloud Console, you must
configure Private Endpoints in your VNet from the Azure Portal to make the
Azure Private Link connection to your Confluent Cloud cluster.
Confluent recommends using a
for setting up Private Link endpoints. This configuration automates the manual steps described below.
In the Confluent Cloud Console, you will find the following information for your Confluent Cloud
cluster under the Cluster Settings section. This information is needed
to configure Azure Private Link for a Dedicated cluster in Azure.
Kafka Bootstrap (in the General tab)
DNS domain Name (in the Networking card)
Zonal DNS Subdomain Names (in the Networking card)
For Confluent Cloud single availability zone clusters, create a single Private Endpoint
to the Confluent Cloud Service Alias. For Confluent Cloud multi-availability zone clusters,
create a Private Endpoint to each of the Confluent Cloud zonal Service Aliases.
Create a Private Endpoint for Confluent Cloud by clicking Create Private Endpoint.
Fill in subscription, resource group, name, and region for the virtual
endpoint and click next. The selected subscription must be the same as
the one registered with Confluent Cloud.
Select the Connect to an Azure resource by resource ID or alias
option, paste in the Confluent Cloud Service Alias and click Next. You can
find the Confluent Cloud Service Aliases in the Networking tab under
Cluster settings in the Confluent Cloud Console.
Fill in virtual network and subnet where the Private Endpoint is to be
Click Review + create. Review the details and click Create to
create the Private Endpoint.
Wait for the Azure deployment to complete, go to the Private Endpoint
resource and verify Private Endpoint connection status is Approved.
Set up DNS records to use Azure Private Endpoints¶
DNS changes must be made to ensure connectivity passes through Azure Private
Link in the supported pattern. Any DNS provider that can ensure DNS is routed
as follows is acceptable. Azure Private DNS Zone (used in this example) is one
From an instance within the VNet, or anywhere the DNS is set up, run the
following to validate Kafka connectivity through Azure Private Link is working
Set an environment variable with the cluster bootstrap URL.
The Bootstrap URL displayed in Confluent Cloud Console includes the port (9092).
The BOOTSTRAP value should include the full hostname, but do not include
the port. This is so that you can run the
openssls_client-connect<host>:<port> command with the required values.
To run the openssls_client-connect command, the -connect option requires
that you specify the host and the port number. For details, see the openssl documentation
for the -connect option
option in the openssl s_client documentation.
If the output returned is -----BEGINCERTIFICATE-----Verifyreturncode:0(ok),
then connectivity to the bootstrap is confirmed.
Log in to the Confluent Cloud CLI with your Confluent Cloud credentials.
List the clusters in your organization.
confluent kafka cluster list
Select the cluster with Azure Private Link you wish to test.
confluent kafka cluster use ...
confluent kafka cluster use lkc-222v1o
Create a cluster API key to authenticate with the cluster.
confluent api-key use R4XPKKUPLYZSHOAT --resource lkc-222v1o
Create a test topic.
confluent kafka topic create test
Start consuming events from the test topic.
confluent kafka topic consume test
Open another terminal tab or window.
Start a producer.
confluent kafka topic produce test
Type anything into the produce tab and hit Enter; press Ctrl+D
or Ctrl+C to stop the producer.
The tab running consume will print what was typed in the tab running produce.
You’re done! The cluster is ready for use.
The bootstrap and broker hostname DNS resolution for Confluent Cloud Cluster
with Private Link is a two-step process:
The bootstrap and broker hostnames have glb as a subdomain in
their domain name (for example, <cluster-subdomain-name>.eastus2.azure.glb.confluent.cloud).
In the first step, Confluent Cloud Global DNS Resolver
returns a CNAME for bootstrap and broker hostnames which doesn’t include
the glb subdomain (for example, <cluster-subdomain-name>.eastus2.azure.confluent.cloud).
In the second step, the CNAME without the “glb” subdomain is resolved to
private endpoints IP addresses using the Private DNS Zone that you configure
by using the previous steps.
Some DNS systems, like Windows DNS service, lack the ability to recursively
resolve the above mentioned two step resolution within a single DNS node.
For such situations, you should use two DNS systems. The first
DNS system sets up separate forwarding rules for a domain with the “glb”
subdomain and a domain without it, and forwards it to the second DNS system.
The second DNS system recursively resolves by forwarding the “glb”
name resolution request to Confluent Cloud Global DNS Resolver, which
creates the “non-glb” name resolution to the Cloud DNS that hosts the Private
DNS Zone as shown previously. Another alternative is to host the “non-glb” DNS records
locally in the second DNS system.